Filebeat Zeek module not reading all events with -once option

Elastic Stack Beats
,
1

I am using Filebeat 8.0.1 inside of a container and trying to utilize it to read through a bunch of Zeek logs formatted in JSON and ending the process when finished. Ending nicely with an error code is critical because I am planning on using this inside of a job that needs to wait for the error code. I am getting weird results. Often when I run it I only get <100 events even though I am expecting thousands. Somtimes I get zero. My theory is that the process is reading the file and closing the process faster than they are being shipped to Elasticsearch. From looking online it seems that there have been issues with this in other modules as well in the past so it may be related to that.

I have tried a ton of options to include:

-M "..input.close_eof=true" to the command line when running it.

Adding it the configuration file as well

  • module: zeek
    connection:
    enabled: true
    var.paths: ["/logs/conn.log"]
    input:
    close_eof: true

I added a shutdown timer which I was hoping would work though it doesn't appear to help. The process ends well within the timer period which I set.

What logs look like:

filebeat@zeek-job-deploy-test-557844f5db-d62zg:~$ cat /logs/* | wc -l
16481
filebeat@zeek-job-deploy-test-557844f5db-d62zg:~$ head /logs/conn.log

What some of my logs look like:
{"ts":1646845926.07097,"uid":"CQzoVWTyWZh1iy5b","id.orig_h":"172.31.2.58","id.orig_p":60942,"id.resp_h":"172.31.20.4","id.resp_p":8080,"proto":"tcp","duration":0.00024890899658203125,"orig_bytes":0,"resp_bytes":0,"conn_state":"SF","missed_bytes":0,"history":"FfA","orig_pkts":4,"orig_ip_bytes":208,"resp_pkts":2,"resp_ip_bytes":104}
{"ts":1646845926.130147,"uid":"CikyGS2yx5pixyEXtj","id.orig_h":"172.31.2.58","id.orig_p":60944,"id.resp_h":"172.31.20.4","id.resp_p":8080,"proto":"tcp","duration":0.00015497207641601563,"orig_bytes":0,"resp_bytes":0,"conn_state":"SF","missed_bytes":0,"history":"FfA","orig_pkts":4,"orig_ip_bytes":208,"resp_pkts":2,"resp_ip_bytes":104}
... more of this.....

Tests using the once option and not using the once option

rm -rf data/registry
filebeat -environment container -once -M "*.*.input.close_eof=true" -d "*"

#Sometimes I get a few events, but I never get all of the events.
#queue":{"acked":0,"max_events":4096}}}

rm -rf data/registry
filebeat -environment container -M "*.*.input.close_eof=true"
#"events":{"acked":6805,"active":0,"batches":97,"total":6805}

rm -rf data/registry
filebeat -environment container
#:{"events":{"acked":6805,"active":0,"batches":97,"total":6805}

My configuration (currently I am only using conn.log for debugging, but plan on using all):

######filebeat.yml#######
filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

filebeat.shutdown_timeout: 20s

processors:
  - add_cloud_metadata: ~
  - add_docker_metadata: ~


#output.file:
#  path: "/tmp/filebeat"
#  filename: filebeat



output.elasticsearch:
  hosts: '${ELASTICSEARCH_HOSTS}'
  username: '${ELASTICSEARCH_USERNAME:}'
  password: '${ELASTICSEARCH_PASSWORD:}'
  ssl.verification_mode: none


##### modules.d/zeek.yml ######
# Module: zeek
# Docs: https://www.elastic.co/guide/en/beats/filebeat/8.0/filebeat-module-zeek.html

- module: zeek
  connection:
    enabled: true
    var.paths: ["/logs/conn.log"]
    input:
      close_eof: true
2

Debug Logs with once option:

filebeat@zeek-job-deploy-test-557844f5db-d62zg:~$ filebeat -environment container -once -M "*.*.input.close_eof=true"  -d "*"
{"log.level":"info","@timestamp":"2022-03-11T13:25:45.251Z","log.origin":{"file.name":"instance/beat.go","file.line":679},"message":"Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:45.251Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":737},"message":"Beat metadata path: /usr/share/filebeat/data/meta.json","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-11T13:25:45.251Z","log.origin":{"file.name":"instance/beat.go","file.line":687},"message":"Beat ID: 51a07b94-eee0-48b0-97f9-14086d6973e4","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-03-11T13:25:48.253Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/provider_aws_ec2.go","file.line":80},"message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers). No token in the metadata request will be used.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:48.253Z","log.logger":"docker","log.origin":{"file.name":"docker/client.go","file.line":49},"message":"Docker client will negotiate the API version on the first request.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:48.253Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/providers.go","file.line":129},"message":"add_cloud_metadata: starting to fetch metadata, timeout=3s","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:48.254Z","log.logger":"add_docker_metadata","log.origin":{"file.name":"add_docker_metadata/add_docker_metadata.go","file.line":88},"message":"add_docker_metadata: docker environment not detected: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.254Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/providers.go","file.line":173},"message":"add_cloud_metadata: timed-out waiting for all responses","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.254Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/providers.go","file.line":132},"message":"add_cloud_metadata: fetchMetadata ran for 3.00114354s","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-11T13:25:51.254Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/add_cloud_metadata.go","file.line":101},"message":"add_cloud_metadata: hosting provider type not detected.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.255Z","log.logger":"processors","log.origin":{"file.name":"processors/processor.go","file.line":120},"message":"Generated new processors: add_cloud_metadata={}, add_docker_metadata=[match_fields=[] match_pids=[process.pid, process.parent.pid]]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.255Z","log.logger":"seccomp","log.origin":{"file.name":"seccomp/seccomp.go","file.line":117},"message":"Loading syscall filter","service.name":"filebeat","seccomp_filter":{"no_new_privs":true,"flag":"tsync","policy":{"default_action":"errno","syscalls":[{"names":["accept","accept4","access","arch_prctl","bind","brk","chmod","chown","clock_gettime","clone","clone3","close","connect","dup","dup2","epoll_create","epoll_create1","epoll_ctl","epoll_pwait","epoll_wait","exit","exit_group","fchdir","fchmod","fchmodat","fchown","fchownat","fcntl","fdatasync","flock","fstat","fstatfs","fsync","ftruncate","futex","getcwd","getdents","getdents64","geteuid","getgid","getpeername","getpid","getppid","getrandom","getrlimit","getrusage","getsockname","getsockopt","gettid","gettimeofday","getuid","inotify_add_watch","inotify_init1","inotify_rm_watch","ioctl","kill","listen","lseek","lstat","madvise","mincore","mkdirat","mmap","mprotect","munmap","nanosleep","newfstatat","open","openat","pipe","pipe2","poll","ppoll","pread64","pselect6","pwrite64","read","readlink","readlinkat","recvfrom","recvmmsg","recvmsg","rename","renameat","rt_sigaction","rt_sigprocmask","rt_sigreturn","sched_getaffinity","sched_yield","sendfile","sendmmsg","sendmsg","sendto","set_robust_list","setitimer","setsockopt","shutdown","sigaltstack","socket","splice","stat","statfs","sysinfo","tgkill","time","tkill","uname","unlink","unlinkat","wait4","waitid","write","writev"],"action":"allow"}]}},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-11T13:25:51.256Z","log.logger":"seccomp","log.origin":{"file.name":"seccomp/seccomp.go","file.line":124},"message":"Syscall filter successfully installed","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-11T13:25:51.256Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1050},"message":"Beat info","service.name":"filebeat","system_info":{"beat":{"path":{"config":"/usr/share/filebeat","data":"/usr/share/filebeat/data","home":"/usr/share/filebeat","logs":"/usr/share/filebeat/logs"},"type":"filebeat","uuid":"51a07b94-eee0-48b0-97f9-14086d6973e4"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-03-11T13:25:51.256Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1059},"message":"Build info","service.name":"filebeat","system_info":{"build":{"commit":"6e9dd49b5da9c045125078bb95be9f0dc27e8263","libbeat":"8.0.1","time":"2022-02-24T15:08:16.000Z","version":"8.0.1"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-03-11T13:25:51.256Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1062},"message":"Go runtime info","service.name":"filebeat","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":8,"version":"go1.17.6"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-03-11T13:25:51.257Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1066},"message":"Host info","service.name":"filebeat","system_info":{"host":{"architecture":"x86_64","boot_time":"2022-03-01T12:51:56Z","containerized":true,"name":"zeek-job-deploy-test-557844f5db-d62zg","ip":["127.0.0.1/8","::1/128","10.42.3.144/32","fe80::4cb9:f9ff:fe1c:ed4a/64"],"kernel_version":"4.18.0-348.el8.x86_64","mac":["4e:b9:f9:1c:ed:4a"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"20.04.3 LTS (Focal Fossa)","major":20,"minor":4,"patch":3,"codename":"focal"},"timezone":"UTC","timezone_offset_sec":0},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-03-11T13:25:51.258Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1095},"message":"Process info","service.name":"filebeat","system_info":{"process":{"capabilities":{"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":null,"effective":null,"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null},"cwd":"/usr/share/filebeat","exe":"/usr/share/filebeat/filebeat","name":"filebeat","pid":3089,"ppid":3056,"seccomp":{"mode":"filter","no_new_privs":true},"start_time":"2022-03-11T13:25:44.380Z"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-03-11T13:25:51.258Z","log.origin":{"file.name":"instance/beat.go","file.line":332},"message":"Setup Beat: filebeat; Version: 8.0.1","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.258Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":360},"message":"Initializing output plugins","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-03-11T13:25:51.259Z","log.logger":"cfgwarn","log.origin":{"file.name":"tlscommon/config.go","file.line":102},"message":"DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-11T13:25:51.259Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":105},"message":"elasticsearch url: https://pcaplogs-es-http:9200","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-03-11T13:25:51.259Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":105},"message":"SSL/TLS verifications disabled.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.259Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/consumer.go","file.line":98},"message":"start pipeline event consumer","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-11T13:25:51.259Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: zeek-job-deploy-test-557844f5db-d62zg","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.259Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/queue_reader.go","file.line":48},"message":"pipeline event consumer queue reader: start","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-11T13:25:51.260Z","log.origin":{"file.name":"fileset/modules.go","file.line":103},"message":"Enabled modules/filesets:  ()","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-11T13:25:51.260Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":142},"message":"Starting metrics logging every 30s","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-11T13:25:51.260Z","log.origin":{"file.name":"instance/beat.go","file.line":498},"message":"filebeat start running.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.261Z","log.logger":"test","log.origin":{"file.name":"registrar/migrate.go","file.line":304},"message":"isFile(/usr/share/filebeat/data/registry) -> false","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.261Z","log.logger":"test","log.origin":{"file.name":"registrar/migrate.go","file.line":304},"message":"isFile() -> false","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.261Z","log.logger":"test","log.origin":{"file.name":"registrar/migrate.go","file.line":297},"message":"isDir(/usr/share/filebeat/data/registry/filebeat) -> true","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.261Z","log.logger":"test","log.origin":{"file.name":"registrar/migrate.go","file.line":304},"message":"isFile(/usr/share/filebeat/data/registry/filebeat/meta.json) -> true","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.261Z","log.logger":"registrar","log.origin":{"file.name":"registrar/migrate.go","file.line":84},"message":"Registry type '1' found","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-11T13:25:51.261Z","log.origin":{"file.name":"memlog/store.go","file.line":119},"message":"Loading data file of '/usr/share/filebeat/data/registry/filebeat' succeeded. Active transaction id=0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-11T13:25:51.261Z","log.origin":{"file.name":"memlog/store.go","file.line":124},"message":"Finished loading transaction log file for '/usr/share/filebeat/data/registry/filebeat'. Active transaction id=0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.261Z","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform request:append","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.262Z","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform request:delete","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.262Z","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform request:set","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.262Z","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform response:append","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.262Z","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform response:delete","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.262Z","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform response:set","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.262Z","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform pagination:append","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.262Z","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform pagination:delete","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.262Z","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform pagination:set","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.262Z","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/encoding.go","file.line":82},"message":"<nil>","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.262Z","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/encoding.go","file.line":83},"message":"<nil>","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.262Z","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/encoding.go","file.line":88},"message":"<nil>","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.262Z","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/encoding.go","file.line":89},"message":"<nil>","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.262Z","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/encoding.go","file.line":90},"message":"<nil>","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-11T13:25:51.262Z","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":109},"message":"States Loaded from registrar: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-11T13:25:51.262Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":71},"message":"Loading Inputs: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.262Z","log.logger":"cfgfile","log.origin":{"file.name":"cfgfile/reload.go","file.line":132},"message":"Checking module configs from: /usr/share/filebeat/modules.d/*.yml","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.262Z","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":140},"message":"Starting Registrar","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.262Z","log.logger":"cfgfile","log.origin":{"file.name":"cfgfile/cfgfile.go","file.line":193},"message":"Load config from file: /usr/share/filebeat/modules.d/zeek.yml","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.263Z","log.logger":"cfgfile","log.origin":{"file.name":"cfgfile/reload.go","file.line":146},"message":"Number of module configs found: 1","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-11T13:25:51.264Z","log.origin":{"file.name":"fileset/modules.go","file.line":103},"message":"Enabled modules/filesets: zeek (connection)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-03-11T13:25:51.265Z","log.logger":"cfgwarn","log.origin":{"file.name":"log/input.go","file.line":89},"message":"DEPRECATED: Log input. Use Filestream input instead.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.265Z","log.logger":"input","log.origin":{"file.name":"log/config.go","file.line":207},"message":"recursive glob enabled","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.266Z","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":185},"message":"exclude_files: [(?-s:.)gz(?-m:$)]. Number of states: 0","service.name":"filebeat","input_id":"3a899329-2af9-4bbb-82e5-3c113f31caba","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.266Z","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":215},"message":"input with previous states loaded: 0","service.name":"filebeat","input_id":"3a899329-2af9-4bbb-82e5-3c113f31caba","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-11T13:25:51.266Z","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":171},"message":"Configured paths: [/logs/conn.log]","service.name":"filebeat","input_id":"3a899329-2af9-4bbb-82e5-3c113f31caba","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-11T13:25:51.266Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":108},"message":"Loading and starting Inputs completed. Enabled inputs: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-11T13:25:51.266Z","log.origin":{"file.name":"cfgfile/reload.go","file.line":164},"message":"Config reloader started","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-11T13:25:51.266Z","log.origin":{"file.name":"beater/filebeat.go","file.line":351},"message":"Running filebeat once. Waiting for completion ...","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-11T13:25:51.266Z","log.origin":{"file.name":"beater/filebeat.go","file.line":353},"message":"All data collection completed. Shutting down.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.266Z","log.logger":"cfgfile","log.origin":{"file.name":"cfgfile/reload.go","file.line":194},"message":"Scan for new config files","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-11T13:25:51.266Z","log.origin":{"file.name":"beater/crawler.go","file.line":148},"message":"Stopping Crawler","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-11T13:25:51.266Z","log.origin":{"file.name":"beater/crawler.go","file.line":158},"message":"Stopping 0 inputs","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.266Z","log.logger":"cfgfile","log.origin":{"file.name":"cfgfile/cfgfile.go","file.line":193},"message":"Load config from file: /usr/share/filebeat/modules.d/zeek.yml","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.267Z","log.logger":"cfgfile","log.origin":{"file.name":"cfgfile/reload.go","file.line":213},"message":"Number of module configs found: 1","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.267Z","log.logger":"reload","log.origin":{"file.name":"cfgfile/list.go","file.line":63},"message":"Starting reload procedure, current runners: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.267Z","log.logger":"reload","log.origin":{"file.name":"cfgfile/list.go","file.line":81},"message":"Start list: 1, Stop list: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-11T13:25:51.268Z","log.origin":{"file.name":"fileset/modules.go","file.line":103},"message":"Enabled modules/filesets: zeek (connection)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.269Z","log.logger":"conditions","log.origin":{"file.name":"conditions/conditions.go","file.line":98},"message":"New condition equals: map[network.transport:0x55dbd4942720]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.270Z","log.logger":"conditions","log.origin":{"file.name":"conditions/conditions.go","file.line":98},"message":"New condition equals: map[network.transport:0x55dbd4942720]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.270Z","log.logger":"processors","log.origin":{"file.name":"processors/processor.go","file.line":120},"message":"Generated new processors: community_id=[target=network.community_id, fields=[source_ip=source.ip, source_port=source.port, destination_ip=destination.ip, destination_port=destination.port, transport_protocol=network.transport, icmp_type=zeek.connection.icmp.type, icmp_code=zeek.connection.icmp.code], seed=0]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.270Z","log.logger":"processors","log.origin":{"file.name":"processors/processor.go","file.line":120},"message":"Generated new processors: community_id=[target=network.community_id, fields=[source_ip=source.ip, source_port=source.port, destination_ip=destination.ip, destination_port=destination.port, transport_protocol=network.transport, icmp_type=icmp.type, icmp_code=icmp.code], seed=0]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.270Z","log.logger":"processors","log.origin":{"file.name":"processors/processor.go","file.line":120},"message":"Generated new processors: drop_fields={\"Fields\":[\"json.orig_bytes\",\"json.resp_bytes\",\"json.tunnel_parents\"],\"IgnoreMissing\":true}, rename=[{From:json To:zeek.connection} {From:zeek.connection.duration To:temp.duration} {From:zeek.connection.id.orig_h To:source.address} {From:zeek.connection.id.orig_p To:source.port} {From:zeek.connection.id.resp_h To:destination.address} {From:zeek.connection.id.resp_p To:destination.port} {From:zeek.connection.proto To:network.transport} {From:zeek.connection.service To:network.protocol} {From:zeek.connection.uid To:zeek.session_id} {From:zeek.connection.orig_ip_bytes To:source.bytes} {From:zeek.connection.resp_ip_bytes To:destination.bytes} {From:zeek.connection.orig_pkts To:source.packets} {From:zeek.connection.resp_pkts To:destination.packets} {From:zeek.connection.conn_state To:zeek.connection.state} {From:zeek.connection.orig_l2_addr To:source.mac} {From:zeek.connection.resp_l2_addr To:destination.mac}], rename=[{From:source.port To:zeek.connection.icmp.type} {From:destination.port To:zeek.connection.icmp.code}], condition=equals: map[network.transport:0x55dbd4942720], convert={\"Fields\":[{\"From\":\"zeek.session_id\",\"To\":\"event.id\",\"Type\":\"[unset]\"},{\"From\":\"source.address\",\"To\":\"source.ip\",\"Type\":\"ip\"},{\"From\":\"destination.address\",\"To\":\"destination.ip\",\"Type\":\"ip\"}],\"Tag\":\"\",\"IgnoreMissing\":true,\"FailOnError\":false,\"Mode\":\"copy\"}, add_fields={\"event\":{\"category\":[\"network\"],\"kind\":\"event\"}}, if equals: map[network.transport:0x55dbd4942720] then community_id=[target=network.community_id, fields=[source_ip=source.ip, source_port=source.port, destination_ip=destination.ip, destination_port=destination.port, transport_protocol=network.transport, icmp_type=zeek.connection.icmp.type, icmp_code=zeek.connection.icmp.code], seed=0] else community_id=[target=network.community_id, fields=[source_ip=source.ip, source_port=source.port, destination_ip=destination.ip, destination_port=destination.port, transport_protocol=network.transport, icmp_type=icmp.type, icmp_code=icmp.code], seed=0], networkDirection=source.ip|destination.ip->network.direction, add_fields={\"ecs\":{\"version\":\"1.12.0\"}}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.270Z","log.logger":"input","log.origin":{"file.name":"log/config.go","file.line":207},"message":"recursive glob enabled","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.271Z","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":185},"message":"exclude_files: [(?-s:.)gz(?-m:$)]. Number of states: 0","service.name":"filebeat","input_id":"2b0a8b5c-4eb1-4201-b386-de9d33b665b4","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.271Z","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":215},"message":"input with previous states loaded: 0","service.name":"filebeat","input_id":"2b0a8b5c-4eb1-4201-b386-de9d33b665b4","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-11T13:25:51.271Z","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":171},"message":"Configured paths: [/logs/conn.log]","service.name":"filebeat","input_id":"2b0a8b5c-4eb1-4201-b386-de9d33b665b4","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.271Z","log.logger":"reload","log.origin":{"file.name":"cfgfile/list.go","file.line":105},"message":"Starting runner: zeek (connection)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-11T13:25:51.271Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":105},"message":"elasticsearch url: https://pcaplogs-es-http:9200","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-03-11T13:25:51.271Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":105},"message":"SSL/TLS verifications disabled.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.271Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":260},"message":"ES Ping(url=https://pcaplogs-es-http:9200)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-03-11T13:25:51.271Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":105},"message":"SSL/TLS verifications disabled.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.279Z","log.logger":"esclientleg","log.origin":{"file.name":"transport/logging.go","file.line":41},"message":"Completed dialing successfully","service.name":"filebeat","network":"tcp","address":"pcaplogs-es-http:9200","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.355Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":283},"message":"Ping status code: 200","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-11T13:25:51.355Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":284},"message":"Attempting to connect to Elasticsearch version 7.17.0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.355Z","log.logger":"modules","log.origin":{"file.name":"fileset/pipelines.go","file.line":67},"message":"Required processors: [{geoip ingest-geoip}]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.355Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":339},"message":"GET https://pcaplogs-es-http:9200/_nodes/ingest  <nil>","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.358Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":339},"message":"GET https://pcaplogs-es-http:9200/_ingest/pipeline/filebeat-8.0.1-zeek-connection-pipeline  <nil>","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.358Z","log.logger":"modules","log.origin":{"file.name":"fileset/pipelines.go","file.line":120},"message":"Pipeline already exists in Elasticsearch.","service.name":"filebeat","pipeline":"filebeat-8.0.1-zeek-connection-pipeline","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-11T13:25:51.358Z","log.origin":{"file.name":"cfgfile/reload.go","file.line":224},"message":"Loading of config files completed.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-11T13:25:51.358Z","log.origin":{"file.name":"cfgfile/reload.go","file.line":227},"message":"Dynamic config reloader stopped","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-11T13:25:51.358Z","log.logger":"reload","log.origin":{"file.name":"cfgfile/list.go","file.line":129},"message":"Stopping 1 runners ...","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.358Z","log.logger":"reload","log.origin":{"file.name":"cfgfile/list.go","file.line":140},"message":"Stopping runner: zeek (connection)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.358Z","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":222},"message":"Start next scan","service.name":"filebeat","input_id":"2b0a8b5c-4eb1-4201-b386-de9d33b665b4","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.359Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/client.go","file.line":158},"message":"client: closing acker","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.359Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/client.go","file.line":163},"message":"client: done closing acker","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.359Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/client.go","file.line":165},"message":"client: unlink from queue","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.359Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/client.go","file.line":187},"message":"client: cancelled 0 events","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.359Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/client.go","file.line":167},"message":"client: done unlink","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.359Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/client.go","file.line":170},"message":"client: closing processors","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.359Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/client.go","file.line":175},"message":"client: done closing processors","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-11T13:25:51.359Z","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":524},"message":"Scan aborted because input stopped.","service.name":"filebeat","input_id":"2b0a8b5c-4eb1-4201-b386-de9d33b665b4","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.359Z","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":286},"message":"input states cleaned up. Before: 0, After: 0, Pending: 0","service.name":"filebeat","input_id":"2b0a8b5c-4eb1-4201-b386-de9d33b665b4","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-11T13:25:51.359Z","log.origin":{"file.name":"input/input.go","file.line":136},"message":"input ticker stopped","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.359Z","log.logger":"reload","log.origin":{"file.name":"cfgfile/list.go","file.line":142},"message":"Stopped runner: zeek (connection)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-11T13:25:51.359Z","log.origin":{"file.name":"beater/crawler.go","file.line":178},"message":"Crawler stopped","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-11T13:25:51.359Z","log.origin":{"file.name":"beater/filebeat.go","file.line":406},"message":"Shutdown output timer started. Waiting for max 20s.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-11T13:25:51.359Z","log.origin":{"file.name":"beater/signalwait.go","file.line":93},"message":"Continue shutdown: All enqueued events being published.","service.name":"filebeat","ecs.version":"1.6.0"}
3

Debug Logs continued

{"log.level":"info","@timestamp":"2022-03-11T13:25:51.359Z","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":132},"message":"Stopping Registrar","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-11T13:25:51.359Z","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":166},"message":"Ending Registrar","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.359Z","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":167},"message":"Stopping Registrar","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-11T13:25:51.359Z","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":137},"message":"Registrar stopped","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-11T13:25:51.364Z","log.origin":{"file.name":"cgroup/util.go","file.line":271},"message":"PID 3089 contains a cgroups V2 path (0::/) but no V2 mountpoint was found.\nThis may be because metricbeat is running inside a container on a hybrid system.\nTo monitor cgroups V2 processess in this way, mount the unified (V2) hierarchy inside\nthe container as /sys/fs/cgroup/unified and start the system module with the hostfs setting.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-11T13:25:51.366Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":192},"message":"Total metrics","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000},"quota":{"us":0}},"id":"/","stats":{"periods":0,"throttled":{"ns":0,"periods":0}}},"cpuacct":{"id":"/","total":{"ns":7756415630}},"memory":{"id":"/","mem":{"limit":{"bytes":9223372036854771712},"usage":{"bytes":51355648}}}},"cpu":{"system":{"ticks":70,"time":{"ms":71}},"total":{"ticks":190,"time":{"ms":196},"value":190},"user":{"ticks":120,"time":{"ms":125}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":9},"info":{"ephemeral_id":"bf091201-65d1-4518-8953-84346dc13672","uptime":{"ms":6173},"version":"8.0.1"},"memstats":{"gc_next":17856224,"memory_alloc":9394704,"memory_sys":34161672,"memory_total":52736416,"rss":125501440},"runtime":{"goroutines":24}},"filebeat":{"events":{"active":0,"added":0,"done":0},"harvester":{"closed":0,"open_files":0,"running":0,"skipped":0,"started":0},"input":{"log":{"files":{"renamed":0,"truncated":0}},"netflow":{"flows":0,"packets":{"dropped":0,"received":0}}}},"libbeat":{"config":{"module":{"running":1,"starts":1,"stops":0},"reloads":1,"scans":1},"output":{"events":{"acked":0,"active":0,"batches":0,"dropped":0,"duplicates":0,"failed":0,"toomany":0,"total":0},"read":{"bytes":0,"errors":0},"type":"elasticsearch","write":{"bytes":0,"errors":0}},"pipeline":{"clients":0,"events":{"active":0,"dropped":0,"failed":0,"filtered":0,"published":0,"retry":0,"total":0},"queue":{"acked":0,"max_events":4096}}},"registrar":{"states":{"cleanup":0,"current":0,"update":0},"writes":{"fail":0,"success":0,"total":0}},"system":{"cpu":{"cores":8},"load":{"1":0.45,"15":0.28,"5":0.32,"norm":{"1":0.0563,"15":0.035,"5":0.04}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-03-11T13:25:51.367Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":193},"message":"Uptime: 6.177349569s","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-11T13:25:51.367Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":160},"message":"Stopping metrics logging.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-11T13:25:51.367Z","log.origin":{"file.name":"instance/beat.go","file.line":504},"message":"filebeat stopped.","service.name":"filebeat","ecs.version":"1.6.0"}
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.