Kafka output not working on Filebeat ECK

Elastic Stack Beats
,
1

I'm using filebeat on ECK. I have kafka output but it's not working as expected. There are no logs in filebeat to show that connection with kafka has been established.
Below is my deployment file

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: filebeat-dev
  namespace: smartbox-dev
spec:
  type: filebeat
  version: 8.2.2
  kibanaRef:
    name: kibana-dev
  config:
    filebeat.inputs:
    - type: container
      paths:
      - /var/log/containers/*.log
      fields:
        kafka_topic: "new_topic_logistic"
    output.kafka:
      codec.json:
        pretty: true
        escape_html: false
      hosts: ["kafka-headless.smartbox-dev.svc.cluster.local:9092"]
      topic: '%{[fields.kafka_topic]}'
      partition.round_robin:
        reachable_only: false
      required_acks: 1
      compression: gzip
      max_message_bytes: 1000000
  daemonSet:
    podTemplate:
      spec:
        dnsPolicy: ClusterFirstWithHostNet
        hostNetwork: true
        securityContext:
          runAsUser: 0
        containers:
        - name: filebeat
          volumeMounts:
          - name: varlogcontainers
            mountPath: /var/log/containers
          - name: varlogpods
            mountPath: /var/log/pods
          - name: varlibdockercontainers
            mountPath: /var/lib/docker/containers
        volumes:
        - name: varlogcontainers
          hostPath:
            path: /var/log/containers
        - name: varlogpods
          hostPath:
            path: /var/log/pods
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers

Below are the logs of filebeat pod

{"log.level":"info","@timestamp":"2022-08-12T04:49:22.491Z","log.origin":{"file.name":"instance/beat.go","file.line":685},"message":"Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-12T04:49:22.491Z","log.origin":{"file.name":"instance/beat.go","file.line":693},"message":"Beat ID: 3db52496-9d6e-4a54-8745-db41ddeddfbf","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-12T04:49:22.492Z","log.logger":"seccomp","log.origin":{"file.name":"seccomp/seccomp.go","file.line":124},"message":"Syscall filter successfully installed","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-12T04:49:22.492Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1063},"message":"Beat info","service.name":"filebeat","system_info":{"beat":{"path":{"config":"/usr/share/filebeat","data":"/usr/share/filebeat/data","home":"/usr/share/filebeat","logs":"/usr/share/filebeat/logs"},"type":"filebeat","uuid":"3db52496-9d6e-4a54-8745-db41ddeddfbf"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-08-12T04:49:22.492Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1072},"message":"Build info","service.name":"filebeat","system_info":{"build":{"commit":"2f1e50cc31b960b1a975f2ebe08bd17be9a5e575","libbeat":"8.2.2","time":"2022-05-25T13:25:33.000Z","version":"8.2.2"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-08-12T04:49:22.493Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1075},"message":"Go runtime info","service.name":"filebeat","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":2,"version":"go1.17.10"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-08-12T04:49:22.494Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1079},"message":"Host info","service.name":"filebeat","system_info":{"host":{"architecture":"x86_64","boot_time":"2022-08-08T08:50:26Z","containerized":true,"name":"ip-192-168-65-143.ap-south-1.compute.internal","ip":["127.0.0.1/8","::1/128","192.168.65.143/18","fe80::f1:d0ff:fee3:3992/64","fe80::28b4:92ff:fed9:5f31/64","fe80::3c32:9dff:fe29:e54e/64","fe80::4fa:19ff:fe3e:6c8a/64","fe80::5471:a6ff:fe34:c250/64","fe80::4c91:84ff:feab:7d2f/64","192.168.106.124/18","fe80::53:57ff:fe98:e728/64","fe80::b4a3:39ff:fe0e:e26f/64","fe80::b8c0:7eff:fea0:572d/64","fe80::7cb1:daff:fead:fc9/64","fe80::a8ce:efff:fe2a:c6b7/64","fe80::9086:39ff:feac:e3f1/64","fe80::6c23:1ff:fe62:890d/64","fe80::5849:65ff:fe62:79c7/64","192.168.96.178/18","fe80::d5:72ff:fe24:bb6a/64"],"kernel_version":"5.4.204-113.362.amzn2.x86_64","mac":["02:f1:d0:e3:39:92","2a:b4:92:d9:5f:31","3e:32:9d:29:e5:4e","06:fa:19:3e:6c:8a","56:71:a6:34:c2:50","4e:91:84:ab:7d:2f","02:53:57:98:e7:28","b6:a3:39:0e:e2:6f","ba:c0:7e:a0:57:2d","7e:b1:da:ad:0f:c9","aa:ce:ef:2a:c6:b7","92:86:39:ac:e3:f1","6e:23:01:62:89:0d","5a:49:65:62:79:c7","02:d5:72:24:bb:6a"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"20.04.4 LTS (Focal Fossa)","major":20,"minor":4,"patch":4,"codename":"focal"},"timezone":"UTC","timezone_offset_sec":0},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-08-12T04:49:22.494Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1108},"message":"Process info","service.name":"filebeat","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null},"cwd":"/usr/share/filebeat","exe":"/usr/share/filebeat/filebeat","name":"filebeat","pid":7,"ppid":1,"seccomp":{"mode":"filter","no_new_privs":true},"start_time":"2022-08-12T04:49:20.100Z"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-08-12T04:49:22.494Z","log.origin":{"file.name":"instance/beat.go","file.line":325},"message":"Setup Beat: filebeat; Version: 8.2.2","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-12T04:49:22.496Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: ip-192-168-65-143.ap-south-1.compute.internal","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-12T04:49:22.497Z","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":108},"message":"Enabled modules/filesets: ","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-08-12T04:49:22.497Z","log.origin":{"file.name":"beater/filebeat.go","file.line":166},"message":"Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-12T04:49:22.497Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":142},"message":"Starting metrics logging every 30s","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-08-12T04:49:22.588Z","log.logger":"cfgwarn","log.origin":{"file.name":"tlscommon/config.go","file.line":102},"message":"DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-12T04:49:22.588Z","log.logger":"kibana","log.origin":{"file.name":"kibana/client.go","file.line":182},"message":"Kibana url: https://kibana-dev-kb-http.smartbox-dev.svc:5601","service.name":"filebeat","ecs.version":"1.6.0"}

There is no error, but no logs are being written on kafka topic. Please help

2

@TiagoQueiroz please help me on this

3

On a quick glance your config looks correct, but the logs you posted are not enough. It seems you collected a very short sample of the logs, there are other initialisation steps missing from those logs, specially the output part.

A couple things you can try:

  • Run Filebeat with log debug enabled
  • Make sure Filebeat's container has got enough resources and is not restarting before it finishes its initialisation steps.
4

Here are the logs after debug enabled

{"log.level":"info","@timestamp":"2022-08-12T08:01:52.746Z","log.origin":{"file.name":"instance/beat.go","file.line":685},"message":"Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:01:52.746Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":743},"message":"Beat metadata path: /usr/share/filebeat/data/meta.json","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-12T08:01:52.838Z","log.origin":{"file.name":"instance/beat.go","file.line":693},"message":"Beat ID: 145accfd-c6c8-4430-b314-fdc0d34ebc7b","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:01:52.838Z","log.logger":"seccomp","log.origin":{"file.name":"seccomp/seccomp.go","file.line":117},"message":"Loading syscall filter","service.name":"filebeat","seccomp_filter":{"no_new_privs":true,"flag":"tsync","policy":{"default_action":"errno","syscalls":[{"names":["accept","accept4","access","arch_prctl","bind","brk","chmod","chown","clock_gettime","clone","clone3","close","connect","dup","dup2","epoll_create","epoll_create1","epoll_ctl","epoll_pwait","epoll_wait","exit","exit_group","fchdir","fchmod","fchmodat","fchown","fchownat","fcntl","fdatasync","flock","fstat","fstatfs","fsync","ftruncate","futex","getcwd","getdents","getdents64","geteuid","getgid","getpeername","getpid","getppid","getrandom","getrlimit","getrusage","getsockname","getsockopt","gettid","gettimeofday","getuid","inotify_add_watch","inotify_init1","inotify_rm_watch","ioctl","kill","listen","lseek","lstat","madvise","mincore","mkdirat","mmap","mprotect","munmap","nanosleep","newfstatat","open","openat","pipe","pipe2","poll","ppoll","pread64","pselect6","pwrite64","read","readlink","readlinkat","recvfrom","recvmmsg","recvmsg","rename","renameat","rseq","rt_sigaction","rt_sigprocmask","rt_sigreturn","sched_getaffinity","sched_yield","sendfile","sendmmsg","sendmsg","sendto","set_robust_list","setitimer","setsockopt","shutdown","sigaltstack","socket","splice","stat","statfs","sysinfo","tgkill","time","tkill","uname","unlink","unlinkat","wait4","waitid","write","writev"],"action":"allow"}]}},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-12T08:01:52.839Z","log.logger":"seccomp","log.origin":{"file.name":"seccomp/seccomp.go","file.line":124},"message":"Syscall filter successfully installed","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-12T08:01:52.839Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1063},"message":"Beat info","service.name":"filebeat","system_info":{"beat":{"path":{"config":"/usr/share/filebeat","data":"/usr/share/filebeat/data","home":"/usr/share/filebeat","logs":"/usr/share/filebeat/logs"},"type":"filebeat","uuid":"145accfd-c6c8-4430-b314-fdc0d34ebc7b"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-08-12T08:01:52.839Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1072},"message":"Build info","service.name":"filebeat","system_info":{"build":{"commit":"2f1e50cc31b960b1a975f2ebe08bd17be9a5e575","libbeat":"8.2.2","time":"2022-05-25T13:25:33.000Z","version":"8.2.2"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-08-12T08:01:52.840Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1075},"message":"Go runtime info","service.name":"filebeat","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":2,"version":"go1.17.10"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-08-12T08:01:52.841Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1079},"message":"Host info","service.name":"filebeat","system_info":{"host":{"architecture":"x86_64","boot_time":"2022-08-09T05:19:58Z","containerized":true,"name":"ip-192-168-98-203.ap-south-1.compute.internal","ip":["127.0.0.1/8","::1/128","192.168.98.203/18","fe80::78:feff:fee6:aafa/64","fe80::708e:2dff:fe8c:53e9/64","fe80::8cda:cfff:fe89:c4dc/64","192.168.112.78/18","fe80::9b:39ff:fec1:d8f6/64","fe80::7cfe:15ff:fe27:e922/64","fe80::e405:5bff:fe53:6cca/64","fe80::b872:bcff:fe9b:6a86/64","fe80::904d:49ff:fe77:518a/64","fe80::88a:23ff:fed0:cf7/64","fe80::487:85ff:fe01:f804/64","fe80::1c4d:fcff:fe4c:a4de/64","fe80::802c:80ff:fe1e:9ef8/64"],"kernel_version":"5.4.204-113.362.amzn2.x86_64","mac":["02:78:fe:e6:aa:fa","72:8e:2d:8c:53:e9","8e:da:cf:89:c4:dc","02:9b:39:c1:d8:f6","7e:fe:15:27:e9:22","e6:05:5b:53:6c:ca","ba:72:bc:9b:6a:86","92:4d:49:77:51:8a","0a:8a:23:d0:0c:f7","06:87:85:01:f8:04","1e:4d:fc:4c:a4:de","82:2c:80:1e:9e:f8"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"20.04.4 LTS (Focal Fossa)","major":20,"minor":4,"patch":4,"codename":"focal"},"timezone":"UTC","timezone_offset_sec":0},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-08-12T08:01:52.841Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1108},"message":"Process info","service.name":"filebeat","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null},"cwd":"/usr/share/filebeat","exe":"/usr/share/filebeat/filebeat","name":"filebeat","pid":7,"ppid":1,"seccomp":{"mode":"filter","no_new_privs":true},"start_time":"2022-08-12T08:01:50.219Z"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-08-12T08:01:52.841Z","log.origin":{"file.name":"instance/beat.go","file.line":325},"message":"Setup Beat: filebeat; Version: 8.2.2","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:01:52.842Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":353},"message":"Initializing output plugins","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:01:52.842Z","log.logger":"kafka","log.origin":{"file.name":"kafka/kafka.go","file.line":48},"message":"initialize kafka output","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:01:52.843Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/consumer.go","file.line":98},"message":"start pipeline event consumer","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-12T08:01:52.843Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: ip-192-168-98-203.ap-south-1.compute.internal","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-12T08:01:52.843Z","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":108},"message":"Enabled modules/filesets: ","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-08-12T08:01:52.843Z","log.origin":{"file.name":"beater/filebeat.go","file.line":166},"message":"Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-12T08:01:52.844Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":142},"message":"Starting metrics logging every 30s","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:01:52.843Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/queue_reader.go","file.line":48},"message":"pipeline event consumer queue reader: start","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-08-12T08:01:52.844Z","log.logger":"cfgwarn","log.origin":{"file.name":"tlscommon/config.go","file.line":102},"message":"DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:01:52.844Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls.go","file.line":172},"message":"Successfully loaded CA certificate: /mnt/elastic-internal/kibana-certs/ca.crt","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-12T08:01:52.844Z","log.logger":"kibana","log.origin":{"file.name":"kibana/client.go","file.line":182},"message":"Kibana url: https://kibana-dev-kb-http.smartbox-dev.svc:5601","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:01:52.844Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls.go","file.line":172},"message":"Successfully loaded CA certificate: /mnt/elastic-internal/kibana-certs/ca.crt","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-12T08:02:22.939Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":184},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000},"quota":{"us":10000}},"id":"/","stats":{"periods":319,"throttled":{"ns":28359530693,"periods":238}}},"cpuacct":{"id":"/","total":{"ns":3189142086}},"memory":{"id":"/","mem":{"limit":{"bytes":209715200},"usage":{"bytes":120475648}}}},"cpu":{"system":{"ticks":120,"time":{"ms":120}},"total":{"ticks":3160,"time":{"ms":3160},"value":0},"user":{"ticks":3040,"time":{"ms":3040}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":9},"info":{"ephemeral_id":"f47eb833-a744-4b79-90a3-72ea6d505623","uptime":{"ms":31106},"version":"8.2.2"},"memstats":{"gc_next":86850624,"memory_alloc":50174752,"memory_sys":105435160,"memory_total":392102040,"rss":201613312},"runtime":{"goroutines":16}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0},"type":"kafka"},"pipeline":{"clients":0,"events":{"active":0},"queue":{"max_events":4096}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":2},"load":{"1":1.36,"15":1.11,"5":1.05,"norm":{"1":0.68,"15":0.555,"5":0.525}}}},"ecs.version":"1.6.0"}}
{"log.level":"debug","@timestamp":"2022-08-12T08:02:26.141Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls.go","file.line":172},"message":"Successfully loaded CA certificate: /mnt/elastic-internal/kibana-certs/ca.crt","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-12T08:02:26.141Z","log.logger":"kibana","log.origin":{"file.name":"kibana/client.go","file.line":182},"message":"Kibana url: https://kibana-dev-kb-http.smartbox-dev.svc:5601","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:02:26.141Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls.go","file.line":172},"message":"Successfully loaded CA certificate: /mnt/elastic-internal/kibana-certs/ca.crt","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:02:27.572Z","log.logger":"dashboards","log.origin":{"file.name":"dashboards/kibana_loader.go","file.line":240},"message":"Initialize the Kibana 8.2.2 loader","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:02:27.572Z","log.logger":"dashboards","log.origin":{"file.name":"dashboards/kibana_loader.go","file.line":240},"message":"Kibana URL https://kibana-dev-kb-http.smartbox-dev.svc:5601","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-12T08:02:52.846Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":184},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpu":{"stats":{"periods":121,"throttled":{"ns":2606889751,"periods":27}}},"cpuacct":{"total":{"ns":749499917}},"memory":{"mem":{"usage":{"bytes":-48848896}}}},"cpu":{"system":{"ticks":250,"time":{"ms":130}},"total":{"ticks":3910,"time":{"ms":750},"value":0},"user":{"ticks":3660,"time":{"ms":620}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"f47eb833-a744-4b79-90a3-72ea6d505623","uptime":{"ms":61105},"version":"8.2.2"},"memstats":{"gc_next":30508528,"memory_alloc":16969048,"memory_sys":12648448,"memory_total":457122872,"rss":152907776},"runtime":{"goroutines":19}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":1.01,"15":1.09,"5":1,"norm":{"1":0.505,"15":0.545,"5":0.5}}}},"ecs.version":"1.6.0"}}
{"log.level":"debug","@timestamp":"2022-08-12T08:02:53.485Z","log.logger":"kibana","log.origin":{"file.name":"kibana/client.go","file.line":378},"message":"Imported multipart file to /api/saved_objects/_import with params map[overwrite:[true]]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:02:53.485Z","log.logger":"dashboards","log.origin":{"file.name":"dashboards/kibana_loader.go","file.line":240},"message":"Importing directory /usr/share/filebeat/kibana/7","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:02:53.485Z","log.logger":"dashboards","log.origin":{"file.name":"dashboards/kibana_loader.go","file.line":240},"message":"Import directory /usr/share/filebeat/kibana/7","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:02:53.486Z","log.logger":"dashboards","log.origin":{"file.name":"dashboards/kibana_loader.go","file.line":240},"message":"Import dashboard from /usr/share/filebeat/kibana/7/dashboard/046212a0-a2a1-11e7-928f-5dbe6f6f5519-ecs.json","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:02:53.486Z","log.logger":"dashboards","log.origin":{"file.name":"dashboards/kibana_loader.go","file.line":240},"message":"Importing dashboard from /usr/share/filebeat/kibana/7/dashboard/046212a0-a2a1-11e7-928f-5dbe6f6f5519-ecs.json","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:03:14.500Z","log.logger":"kibana","log.origin":{"file.name":"kibana/client.go","file.line":378},"message":"Imported multipart file to /api/saved_objects/_import with params map[overwrite:[true]]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:03:14.500Z","log.logger":"dashboards","log.origin":{"file.name":"dashboards/kibana_loader.go","file.line":240},"message":"Import dashboard from /usr/share/filebeat/kibana/7/dashboard/0b3dba40-f341-11ea-a3fd-1b45ec532bb3.json","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:03:14.500Z","log.logger":"dashboards","log.origin":{"file.name":"dashboards/kibana_loader.go","file.line":240},"message":"Importing dashboard from /usr/share/filebeat/kibana/7/dashboard/0b3dba40-f341-11ea-a3fd-1b45ec532bb3.json","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-12T08:03:22.848Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":184},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpu":{"stats":{"periods":8}},"cpuacct":{"total":{"ns":12472404}},"memory":{"mem":{"usage":{"bytes":28672}}}},"cpu":{"system":{"ticks":250},"total":{"ticks":3910,"value":0},"user":{"ticks":3660}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"f47eb833-a744-4b79-90a3-72ea6d505623","uptime":{"ms":91105},"version":"8.2.2"},"memstats":{"gc_next":30508528,"memory_alloc":17761664,"memory_total":457915488,"rss":152907776},"runtime":{"goroutines":19}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.77,"15":1.06,"5":0.93,"norm":{"1":0.385,"15":0.53,"5":0.465}}}},"ecs.version":"1.6.0"}}
{"log.level":"debug","@timestamp":"2022-08-12T08:03:33.508Z","log.logger":"kibana","log.origin":{"file.name":"kibana/client.go","file.line":378},"message":"Imported multipart file to /api/saved_objects/_import with params map[overwrite:[true]]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:03:33.508Z","log.logger":"dashboards","log.origin":{"file.name":"dashboards/kibana_loader.go","file.line":240},"message":"Import dashboard from /usr/share/filebeat/kibana/7/dashboard/0d3f2380-fa78-11e6-ae9b-81e5311e8cab-ecs.json","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:03:33.508Z","log.logger":"dashboards","log.origin":{"file.name":"dashboards/kibana_loader.go","file.line":240},"message":"Importing dashboard from /usr/share/filebeat/kibana/7/dashboard/0d3f2380-fa78-11e6-ae9b-81e5311e8cab-ecs.json","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:03:45.422Z","log.logger":"kibana","log.origin":{"file.name":"kibana/client.go","file.line":378},"message":"Imported multipart file to /api/saved_objects/_import with params map[overwrite:[true]]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:03:45.422Z","log.logger":"dashboards","log.origin":{"file.name":"dashboards/kibana_loader.go","file.line":240},"message":"Import dashboard from /usr/share/filebeat/kibana/7/dashboard/0f559cc0-f0d5-11e9-90ec-112a988266d5.json","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:03:45.422Z","log.logger":"dashboards","log.origin":{"file.name":"dashboards/kibana_loader.go","file.line":240},"message":"Importing dashboard from /usr/share/filebeat/kibana/7/dashboard/0f559cc0-f0d5-11e9-90ec-112a988266d5.json","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-12T08:03:52.847Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":184},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpu":{"stats":{"periods":10}},"cpuacct":{"total":{"ns":14257101}},"memory":{"mem":{"usage":{"bytes":36864}}}},"cpu":{"system":{"ticks":250},"total":{"ticks":3930,"time":{"ms":20},"value":0},"user":{"ticks":3680,"time":{"ms":20}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":9},"info":{"ephemeral_id":"f47eb833-a744-4b79-90a3-72ea6d505623","uptime":{"ms":121106},"version":"8.2.2"},"memstats":{"gc_next":30508528,"memory_alloc":19406672,"memory_total":459560496,"rss":152907776},"runtime":{"goroutines":17}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.87,"15":1.06,"5":0.95,"norm":{"1":0.435,"15":0.53,"5":0.475}}}},"ecs.version":"1.6.0"}}
{"log.level":"debug","@timestamp":"2022-08-12T08:03:54.745Z","log.logger":"kibana","log.origin":{"file.name":"kibana/client.go","file.line":378},"message":"Imported multipart file to /api/saved_objects/_import with params map[overwrite:[true]]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:03:54.745Z","log.logger":"dashboards","log.origin":{"file.name":"dashboards/kibana_loader.go","file.line":240},"message":"Import dashboard from /usr/share/filebeat/kibana/7/dashboard/15503340-4488-11ea-ad63-791a5dc86f10.json","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:03:54.745Z","log.logger":"dashboards","log.origin":{"file.name":"dashboards/kibana_loader.go","file.line":240},"message":"Importing dashboard from /usr/share/filebeat/kibana/7/dashboard/15503340-4488-11ea-ad63-791a5dc86f10.json","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:04:07.127Z","log.logger":"kibana","log.origin":{"file.name":"kibana/client.go","file.line":378},"message":"Imported multipart file to /api/saved_objects/_import with params map[overwrite:[true]]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:04:07.127Z","log.logger":"dashboards","log.origin":{"file.name":"dashboards/kibana_loader.go","file.line":240},"message":"Import dashboard from /usr/share/filebeat/kibana/7/dashboard/158be870-87f4-11e7-ad9c-db80de0bf8d3-ecs.json","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:04:07.127Z","log.logger":"dashboards","log.origin":{"file.name":"dashboards/kibana_loader.go","file.line":240},"message":"Importing dashboard from /usr/share/filebeat/kibana/7/dashboard/158be870-87f4-11e7-ad9c-db80de0bf8d3-ecs.json","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:04:13.376Z","log.logger":"kibana","log.origin":{"file.name":"kibana/client.go","file.line":378},"message":"Imported multipart file to /api/saved_objects/_import with params map[overwrite:[true]]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:04:13.376Z","log.logger":"dashboards","log.origin":{"file.name":"dashboards/kibana_loader.go","file.line":240},"message":"Import dashboard from /usr/share/filebeat/kibana/7/dashboard/161855f0-ff6a-11e8-93c5-d5ecd1b3e307-ecs.json","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:04:13.376Z","log.logger":"dashboards","log.origin":{"file.name":"dashboards/kibana_loader.go","file.line":240},"message":"Importing dashboard from /usr/share/filebeat/kibana/7/dashboard/161855f0-ff6a-11e8-93c5-d5ecd1b3e307-ecs.json","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-12T08:04:22.847Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":184},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpu":{"stats":{"periods":10}},"cpuacct":{"total":{"ns":17402895}},"memory":{"mem":{"usage":{"bytes":61440}}}},"cpu":{"system":{"ticks":250},"total":{"ticks":3940,"time":{"ms":10},"value":0},"user":{"ticks":3690,"time":{"ms":10}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":9},"info":{"ephemeral_id":"f47eb833-a744-4b79-90a3-72ea6d505623","uptime":{"ms":151107},"version":"8.2.2"},"memstats":{"gc_next":30508528,"memory_alloc":20947752,"memory_total":461101576,"rss":152907776},"runtime":{"goroutines":17}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.6,"15":1.04,"5":0.88,"norm":{"1":0.3,"15":0.52,"5":0.44}}}},"ecs.version":"1.6.0"}}
{"log.level":"debug","@timestamp":"2022-08-12T08:04:39.094Z","log.logger":"kibana","log.origin":{"file.name":"kibana/client.go","file.line":378},"message":"Imported multipart file to /api/saved_objects/_import with params map[overwrite:[true]]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:04:39.094Z","log.logger":"dashboards","log.origin":{"file.name":"dashboards/kibana_loader.go","file.line":240},"message":"Import dashboard from /usr/share/filebeat/kibana/7/dashboard/26309570-2419-11e7-a83b-d5f4cebac9ff-ecs.json","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:04:39.094Z","log.logger":"dashboards","log.origin":{"file.name":"dashboards/kibana_loader.go","file.line":240},"message":"Importing dashboard from /usr/share/filebeat/kibana/7/dashboard/26309570-2419-11e7-a83b-d5f4cebac9ff-ecs.json","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-12T08:04:52.847Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":184},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpu":{"stats":{"periods":20,"throttled":{"ns":84385382,"periods":1}}},"cpuacct":{"total":{"ns":73897700}},"memory":{"mem":{"usage":{"bytes":-14700544}}}},"cpu":{"system":{"ticks":270,"time":{"ms":20}},"total":{"ticks":4020,"time":{"ms":80},"value":0},"user":{"ticks":3750,"time":{"ms":60}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":9},"info":{"ephemeral_id":"f47eb833-a744-4b79-90a3-72ea6d505623","uptime":{"ms":181106},"version":"8.2.2"},"memstats":{"gc_next":24849536,"memory_alloc":12931600,"memory_total":462089720,"rss":138383360},"runtime":{"goroutines":17}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.8,"15":1.04,"5":0.9,"norm":{"1":0.4,"15":0.52,"5":0.45}}}},"ecs.version":"1.6.0"}}
{"log.level":"debug","@timestamp":"2022-08-12T08:05:09.194Z","log.logger":"dashboards","log.origin":{"file.name":"dashboards/kibana_loader.go","file.line":240},"message":"Import dashboard from /usr/share/filebeat/kibana/7/dashboard/2713ee40-f3b1-11ea-ba07-c1efedbf0bf9.json","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:05:09.195Z","log.logger":"dashboards","log.origin":{"file.name":"dashboards/kibana_loader.go","file.line":240},"message":"Importing dashboard from /usr/share/filebeat/kibana/7/dashboard/2713ee40-f3b1-11ea-ba07-c1efedbf0bf9.json","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-12T08:05:22.847Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":184},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpu":{"stats":{"periods":6}},"cpuacct":{"total":{"ns":13548492}},"memory":{"mem":{"usage":{"bytes":172032}}}},"cpu":{"system":{"ticks":280,"time":{"ms":10}},"total":{"ticks":4030,"time":{"ms":10},"value":0},"user":{"ticks":3750}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":9},"info":{"ephemeral_id":"f47eb833-a744-4b79-90a3-72ea6d505623","uptime":{"ms":211104},"version":"8.2.2"},"memstats":{"gc_next":24849536,"memory_alloc":14672112,"memory_sys":262144,"memory_total":463830232,"rss":138383360},"runtime":{"goroutines":17}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.48,"15":1,"5":0.82,"norm":{"1":0.24,"15":0.5,"5":0.41}}}},"ecs.version":"1.6.0"}}
{"log.level":"debug","@timestamp":"2022-08-12T08:05:36.375Z","log.logger":"kibana","log.origin":{"file.name":"kibana/client.go","file.line":378},"message":"Imported multipart file to /api/saved_objects/_import with params map[overwrite:[true]]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:05:36.375Z","log.logger":"dashboards","log.origin":{"file.name":"dashboards/kibana_loader.go","file.line":240},"message":"Import dashboard from /usr/share/filebeat/kibana/7/dashboard/277876d0-fa2c-11e6-bbd3-29c986c96e5a-ecs.json","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:05:36.375Z","log.logger":"dashboards","log.origin":{"file.name":"dashboards/kibana_loader.go","file.line":240},"message":"Importing dashboard from /usr/share/filebeat/kibana/7/dashboard/277876d0-fa2c-11e6-bbd3-29c986c96e5a-ecs.json","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:05:42.556Z","log.logger":"kibana","log.origin":{"file.name":"kibana/client.go","file.line":378},"message":"Imported multipart file to /api/saved_objects/_import with params map[overwrite:[true]]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:05:42.556Z","log.logger":"dashboards","log.origin":{"file.name":"dashboards/kibana_loader.go","file.line":240},"message":"Import dashboard from /usr/share/filebeat/kibana/7/dashboard/3560d580-aa34-11e8-9c06-877f0445e3e0-ecs.json","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:05:42.556Z","log.logger":"dashboards","log.origin":{"file.name":"dashboards/kibana_loader.go","file.line":240},"message":"Importing dashboard from /usr/share/filebeat/kibana/7/dashboard/3560d580-aa34-11e8-9c06-877f0445e3e0-ecs.json","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-12T08:05:52.846Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":184},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpu":{"stats":{"periods":6}},"cpuacct":{"total":{"ns":10405680}},"memory":{"mem":{"usage":{"bytes":57344}}}},"cpu":{"system":{"ticks":290,"time":{"ms":10}},"total":{"ticks":4040,"time":{"ms":10},"value":0},"user":{"ticks":3750}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":9},"info":{"ephemeral_id":"f47eb833-a744-4b79-90a3-72ea6d505623","uptime":{"ms":241105},"version":"8.2.2"},"memstats":{"gc_next":24849536,"memory_alloc":15972008,"memory_total":465130128,"rss":138383360},"runtime":{"goroutines":17}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.29,"15":0.97,"5":0.74,"norm":{"1":0.145,"15":0.485,"5":0.37}}}},"ecs.version":"1.6.0"}}
{"log.level":"debug","@timestamp":"2022-08-12T08:05:54.024Z","log.logger":"kibana","log.origin":{"file.name":"kibana/client.go","file.line":378},"message":"Imported multipart file to /api/saved_objects/_import with params map[overwrite:[true]]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:05:54.024Z","log.logger":"dashboards","log.origin":{"file.name":"dashboards/kibana_loader.go","file.line":240},"message":"Import dashboard from /usr/share/filebeat/kibana/7/dashboard/3af47420-3e7b-11ea-bb0a-69c3ca1d410f.json","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:05:54.024Z","log.logger":"dashboards","log.origin":{"file.name":"dashboards/kibana_loader.go","file.line":240},"message":"Importing dashboard from /usr/share/filebeat/kibana/7/dashboard/3af47420-3e7b-11ea-bb0a-69c3ca1d410f.json","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:06:00.266Z","log.logger":"kibana","log.origin":{"file.name":"kibana/client.go","file.line":378},"message":"Imported multipart file to /api/saved_objects/_import with params map[overwrite:[true]]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:06:00.266Z","log.logger":"dashboards","log.origin":{"file.name":"dashboards/kibana_loader.go","file.line":240},"message":"Import dashboard from /usr/share/filebeat/kibana/7/dashboard/41e84340-ec20-11e9-90ec-112a988266d5.json","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:06:00.267Z","log.logger":"dashboards","log.origin":{"file.name":"dashboards/kibana_loader.go","file.line":240},"message":"Importing dashboard from /usr/share/filebeat/kibana/7/dashboard/41e84340-ec20-11e9-90ec-112a988266d5.json","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:06:11.752Z","log.logger":"kibana","log.origin":{"file.name":"kibana/client.go","file.line":378},"message":"Imported multipart file to /api/saved_objects/_import with params map[overwrite:[true]]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:06:11.752Z","log.logger":"dashboards","log.origin":{"file.name":"dashboards/kibana_loader.go","file.line":240},"message":"Import dashboard from /usr/share/filebeat/kibana/7/dashboard/4278ad30-fe16-11e7-a3b0-d13028918f9f-ecs.json","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:06:11.752Z","log.logger":"dashboards","log.origin":{"file.name":"dashboards/kibana_loader.go","file.line":240},"message":"Importing dashboard from /usr/share/filebeat/kibana/7/dashboard/4278ad30-fe16-11e7-a3b0-d13028918f9f-ecs.json","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-12T08:06:22.847Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":184},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpu":{"stats":{"periods":10}},"cpuacct":{"total":{"ns":23109656}},"memory":{"mem":{"usage":{"bytes":110592}}}},"cpu":{"system":{"ticks":300,"time":{"ms":10}},"total":{"ticks":4070,"time":{"ms":30},"value":0},"user":{"ticks":3770,"time":{"ms":20}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":9},"info":{"ephemeral_id":"f47eb833-a744-4b79-90a3-72ea6d505623","uptime":{"ms":271104},"version":"8.2.2"},"memstats":{"gc_next":24849536,"memory_alloc":18400768,"memory_total":467558888,"rss":138383360},"runtime":{"goroutines":17}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.69,"15":0.98,"5":0.78,"norm":{"1":0.345,"15":0.49,"5":0.39}}}},"ecs.version":"1.6.0"}}
{"log.level":"debug","@timestamp":"2022-08-12T08:06:33.654Z","log.logger":"kibana","log.origin":{"file.name":"kibana/client.go","file.line":378},"message":"Imported multipart file to /api/saved_objects/_import with params map[overwrite:[true]]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-08-12T08:06:33.654Z","log.logger":"dashboards","log.origin":

The issue is I dont think the kafka output part is working, there is no log coming that says connection to kafka server established

5

About the resources you mentioned, you mean to say the ram and disk?

6

I meant CPU and Ram. Because you posted such a small set of logs, I was wondering if the container had re-started during its startup.

I see in your logs the Kafka output being initialised:

{"log.level":"debug","@timestamp":"2022-08-12T08:01:52.842Z","log.logger":"kafka","log.origin":{"file.name":"kafka/kafka.go","file.line":48},"message":"initialize kafka output","service.name":"filebeat","ecs.version":"1.6.0"}

Filebeat will only connect to Kafka once it has got events to send. Make sure Filebeat has got data to ingest. There were no logs about Filebeat starting the inputs/ingesting data.

Another thing you can try is to test the connection to the output. You can ssh into the container and run filebeat test output. Just make sure it can access its configuration. Running this command won't affect the current instance of Filebeat that is already running.

7

I ran the command by going inside the pod. here is the output

root@ip-192-168-188-192:/usr/share/filebeat# filebeat test output
elasticsearch: http://elasticsearch:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... ERROR lookup elasticsearch on 10.100.0.10:53: no such host

I don't know why it's trying to connect with elasticsearch, I haven't given any elasticsearch output.
And also yes the containers restarted quite a few times during startup.

NAME                                         READY   STATUS    RESTARTS         AGE

elastic-operator-0                           1/1     Running   0                4d
elasticsearch-dev-es-es-master-data-dev-0    1/1     Running   0                3d4h
elasticsearch-dev-es-es-master-data-dev-1    1/1     Running   0                4d
elasticsearch-dev-es-es-master-data-dev-2    1/1     Running   0                4d
filebeat-dev-beat-filebeat-4xzs5             1/1     Running   14 (4m59s ago)   86m
filebeat-dev-beat-filebeat-dr5t6             1/1     Running   12 (86s ago)     86m
filebeat-dev-beat-filebeat-xcdpl             1/1     Running   8 (17m ago)      86m
8

In that case it seems Filebeat used the default config and ECK is passing the configuration in a different manner, hence the filebeat test output is not testing the Kafka output.

Anyway, if there is any issues communicating with Kafka that should be in the logs. Based on your logs the right output is being initialised.

Did you see any "published events" in the logs? Look for log lines containing Publish event:.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.