kubectl logs filebeat-filebeat-bl6x6 -f
2020-02-09T00:55:31.955Z INFO instance/beat.go:610 Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2020-02-09T00:55:31.957Z INFO instance/beat.go:618 Beat ID: 09f85cbb-0718-4c37-ae96-794f5cecfa9b
2020-02-09T00:55:31.960Z INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed
2020-02-09T00:55:31.960Z INFO [beat] instance/beat.go:941 Beat info {"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "09f85cbb-0718-4c37-ae96-794f5cecfa9b"}}}
2020-02-09T00:55:31.961Z INFO [beat] instance/beat.go:950 Build info {"system_info": {"build": {"commit": "a9c141434cd6b25d7a74a9c770be6b70643dc767", "libbeat": "7.5.2", "time": "2020-01-15T11:13:22.000Z", "version": "7.5.2"}}}
2020-02-09T00:55:31.961Z INFO [beat] instance/beat.go:953 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":3,"version":"go1.12.12"}}}
2020-02-09T00:55:31.967Z INFO [beat] instance/beat.go:957 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-02-07T07:07:31Z","containerized":true,"name":"filebeat-filebeat-bl6x6","ip":["127.0.0.1/8","::1/128","10.244.1.94/24","fe80::841e:91ff:fe64:ad3d/64"],"kernel_version":"4.19.76-linuxkit","mac":["86:1e:91:64:ad:3d"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":7,"patch":1908,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
2020-02-09T00:55:31.967Z INFO [beat] instance/beat.go:986 Process info {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-02-09T00:55:31.580Z"}}}
2020-02-09T00:55:31.967Z INFO instance/beat.go:297 Setup Beat: filebeat; Version: 7.5.2
2020-02-09T00:55:31.969Z INFO [publisher] pipeline/module.go:97 Beat name: filebeat-filebeat-bl6x6
2020-02-09T00:55:31.975Z WARN beater/filebeat.go:152 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-02-09T00:55:31.977Z INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2020-02-09T00:55:31.977Z INFO instance/beat.go:429 filebeat start running.
2020-02-09T00:55:31.978Z INFO registrar/registrar.go:145 Loading registrar data from /usr/share/filebeat/data/registry/filebeat/data.json
2020-02-09T00:55:31.979Z INFO registrar/registrar.go:152 States Loaded from registrar: 12
2020-02-09T00:55:31.979Z WARN beater/filebeat.go:368 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-02-09T00:55:31.979Z INFO crawler/crawler.go:72 Loading Inputs: 0
2020-02-09T00:55:31.979Z INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 0
2020-02-09T00:55:31.979Z WARN [cfgwarn] kubernetes/kubernetes.go:59 BETA: The kubernetes autodiscover is beta
2020-02-09T00:55:31.984Z INFO kubernetes/util.go:94 kubernetes: Using pod name filebeat-filebeat-bl6x6 and namespace default to discover kubernetes node
2020-02-09T00:55:32.042Z INFO kubernetes/util.go:100 kubernetes: Using node kind-worker discovered by in cluster pod node query
2020-02-09T00:55:32.043Z INFO [autodiscover] autodiscover/autodiscover.go:104 Starting autodiscover manager
2020-02-09T00:55:32.147Z INFO log/input.go:152 Configured paths: [/var/lib/docker/containers/49ebfad7ad3775ba66e6e775bc089fa27b79fe59efa016e695791f6c799e9945/*-json.log]
2020-02-09T00:55:32.148Z INFO input/input.go:114 Starting input of type: container; ID: 9137770389487786806
2020-02-09T00:55:32.149Z INFO log/input.go:152 Configured paths: [/var/lib/docker/containers/406c310c7e514bd2e603105c1946a5944cc8f7695923997a38523897415e788d/*-json.log]
2020-02-09T00:55:32.150Z INFO input/input.go:114 Starting input of type: container; ID: 15042780179735339830
2020-02-09T00:55:32.151Z INFO log/input.go:152 Configured paths: [/var/lib/docker/containers/5dfb2f8cf7478e9ae993ffa6d182eaff4e8ed3d8e24094b0233ee3711981befb/*-json.log]
2020-02-09T00:55:32.151Z INFO input/input.go:114 Starting input of type: container; ID: 11430920626231009588
2020-02-09T00:55:32.154Z INFO log/input.go:152 Configured paths: [/var/lib/docker/containers/fb0fed987f7ac758030deab2fa09e96359c7e51dd4df0695fe6213dc2af90f45/*-json.log]
2020-02-09T00:55:32.154Z INFO input/input.go:114 Starting input of type: container; ID: 15751043935070790471
2020-02-09T00:55:32.156Z INFO log/input.go:152 Configured paths: [/var/lib/docker/containers/e95db206a1050d819a20f9c5509ff158309a4b00ce2d0a34d5aea9271389bbb6/*-json.log]
2020-02-09T00:55:32.156Z INFO input/input.go:114 Starting input of type: container; ID: 9433795024465231413
2020-02-09T00:55:32.156Z INFO log/input.go:152 Configured paths: [/var/lib/docker/containers/a52f0d70b9db6f884e96ff57f66b005422f90139d6dbf61b36073a71aee4263b/*-json.log]
2020-02-09T00:55:32.156Z INFO input/input.go:114 Starting input of type: container; ID: 18412748010883190238
2020-02-09T00:55:32.158Z INFO log/input.go:152 Configured paths: [/var/lib/docker/containers/2b51132d5c1a1bc42395cad57778cc1ebe62b97c7db526ba8ead29c3f08c685d/*-json.log]
2020-02-09T00:55:32.158Z INFO input/input.go:114 Starting input of type: container; ID: 13976086098406123311
2020-02-09T00:55:32.429Z INFO log/input.go:152 Configured paths: [/var/lib/docker/containers/dbdd578dbe58089c32bfaf78c0c32619f46f48646e67eb6276d10fcadc456490/*-json.log]
2020-02-09T00:55:32.429Z INFO input/input.go:114 Starting input of type: container; ID: 14861032766883028300
2020-02-09T00:55:43.985Z INFO input/input.go:149 input ticker stopped
2020-02-09T00:55:43.985Z INFO input/input.go:167 Stopping Input: 14861032766883028300
2020-02-09T00:55:43.986Z INFO log/input.go:152 Configured paths: [/var/lib/docker/containers/dbdd578dbe58089c32bfaf78c0c32619f46f48646e67eb6276d10fcadc456490/*-json.log]
2020-02-09T00:55:43.986Z INFO input/input.go:114 Starting input of type: container; ID: 14861032766883028300
2020-02-09T00:56:01.947Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":140,"time":{"ms":147}},"total":{"ticks":330,"time":{"ms":340},"value":330},"user":{"ticks":190,"time":{"ms":193}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":8},"info":{"ephemeral_id":"eccb85cf-2921-47c4-bda2-bf6c0aee80c0","uptime":{"ms":30111}},"memstats":{"gc_next":10350064,"memory_alloc":5516560,"memory_total":15688664,"rss":49332224},"runtime":{"goroutines":75}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"kafka"},"pipeline":{"clients":8,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":3},"load":{"1":5.82,"15":13.8,"5":9.63,"norm":{"1":1.94,"15":4.6,"5":3.21}}}}}}
2020-02-09T00:56:04.583Z INFO beater/filebeat.go:443 Stopping filebeat
2020-02-09T00:56:04.583Z INFO [autodiscover] cfgfile/list.go:118 Stopping 8 runners ...
2020-02-09T00:56:04.583Z INFO input/input.go:149 input ticker stopped
2020-02-09T00:56:04.584Z INFO input/input.go:167 Stopping Input: 15042780179735339830
2020-02-09T00:56:04.584Z INFO input/input.go:149 input ticker stopped
2020-02-09T00:56:04.584Z INFO input/input.go:167 Stopping Input: 11430920626231009588
2020-02-09T00:56:04.584Z INFO input/input.go:149 input ticker stopped
2020-02-09T00:56:04.584Z INFO input/input.go:167 Stopping Input: 15751043935070790471
2020-02-09T00:56:04.584Z INFO input/input.go:149 input ticker stopped
2020-02-09T00:56:04.584Z INFO input/input.go:167 Stopping Input: 9433795024465231413
2020-02-09T00:56:04.584Z INFO input/input.go:149 input ticker stopped
2020-02-09T00:56:04.584Z INFO input/input.go:167 Stopping Input: 18412748010883190238
2020-02-09T00:56:04.584Z INFO input/input.go:149 input ticker stopped
2020-02-09T00:56:04.584Z INFO input/input.go:167 Stopping Input: 13976086098406123311
2020-02-09T00:56:04.584Z INFO input/input.go:149 input ticker stopped
2020-02-09T00:56:04.584Z INFO input/input.go:167 Stopping Input: 14861032766883028300
2020-02-09T00:56:04.584Z INFO input/input.go:149 input ticker stopped
2020-02-09T00:56:04.584Z INFO input/input.go:167 Stopping Input: 9137770389487786806
2020-02-09T00:56:04.584Z INFO [autodiscover] autodiscover/autodiscover.go:292 Stopped autodiscover manager
2020-02-09T00:56:04.584Z INFO crawler/crawler.go:139 Stopping Crawler
2020-02-09T00:56:04.584Z INFO crawler/crawler.go:149 Stopping 0 inputs
2020-02-09T00:56:04.585Z INFO crawler/crawler.go:165 Crawler stopped
2020-02-09T00:56:04.585Z INFO registrar/registrar.go:367 Stopping Registrar
2020-02-09T00:56:04.585Z INFO registrar/registrar.go:293 Ending Registrar
2020-02-09T00:56:04.597Z INFO [monitoring] log/log.go:153 Total non-zero metrics {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":150,"time":{"ms":157}},"total":{"ticks":340,"time":{"ms":355},"value":340},"user":{"ticks":190,"time":{"ms":198}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":8},"info":{"ephemeral_id":"eccb85cf-2921-47c4-bda2-bf6c0aee80c0","uptime":{"ms":32763}},"memstats":{"gc_next":10350064,"memory_alloc":5753744,"memory_total":15925848,"rss":49332224},"runtime":{"goroutines":13}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"kafka"},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":12},"writes":{"success":1,"total":1}},"system":{"cpu":{"cores":3},"load":{"1":5.82,"15":13.8,"5":9.63,"norm":{"1":1.94,"15":4.6,"5":3.21}}}}}}
2020-02-09T00:56:04.599Z INFO [monitoring] log/log.go:154 Uptime: 32.7656174s
2020-02-09T00:56:04.600Z INFO [monitoring] log/log.go:131 Stopping metrics logging.
2020-02-09T00:56:04.600Z INFO instance/beat.go:435 filebeat stopped.
Here is my filebeat confg file
filebeat.autodiscover:
providers:
- type: kubernetes
hints.enabled: true
processors:
- add_kubernetes_metadata:
in_cluster: true
output.kafka:
codec.format:
string: '%{[@timestamp]} %{[message]}'
hosts: ["kafka:9092"]
topic: 'log'
partition.round_robin:
reachable_only: false
required_acks: 1
compression: gzip
max_message_bytes: 1000000
logging.level: info