Filebeat pod stopped, crashloopbackoff

izazahamed.babji · February 9, 2020, 1:53am

kubectl logs filebeat-filebeat-bl6x6 -f
2020-02-09T00:55:31.955Z	INFO	instance/beat.go:610	Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2020-02-09T00:55:31.957Z	INFO	instance/beat.go:618	Beat ID: 09f85cbb-0718-4c37-ae96-794f5cecfa9b
2020-02-09T00:55:31.960Z	INFO	[seccomp]	seccomp/seccomp.go:124	Syscall filter successfully installed
2020-02-09T00:55:31.960Z	INFO	[beat]	instance/beat.go:941	Beat info	{"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "09f85cbb-0718-4c37-ae96-794f5cecfa9b"}}}
2020-02-09T00:55:31.961Z	INFO	[beat]	instance/beat.go:950	Build info	{"system_info": {"build": {"commit": "a9c141434cd6b25d7a74a9c770be6b70643dc767", "libbeat": "7.5.2", "time": "2020-01-15T11:13:22.000Z", "version": "7.5.2"}}}
2020-02-09T00:55:31.961Z	INFO	[beat]	instance/beat.go:953	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":3,"version":"go1.12.12"}}}
2020-02-09T00:55:31.967Z	INFO	[beat]	instance/beat.go:957	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-02-07T07:07:31Z","containerized":true,"name":"filebeat-filebeat-bl6x6","ip":["127.0.0.1/8","::1/128","10.244.1.94/24","fe80::841e:91ff:fe64:ad3d/64"],"kernel_version":"4.19.76-linuxkit","mac":["86:1e:91:64:ad:3d"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":7,"patch":1908,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
2020-02-09T00:55:31.967Z	INFO	[beat]	instance/beat.go:986	Process info	{"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-02-09T00:55:31.580Z"}}}
2020-02-09T00:55:31.967Z	INFO	instance/beat.go:297	Setup Beat: filebeat; Version: 7.5.2
2020-02-09T00:55:31.969Z	INFO	[publisher]	pipeline/module.go:97	Beat name: filebeat-filebeat-bl6x6
2020-02-09T00:55:31.975Z	WARN	beater/filebeat.go:152	Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-02-09T00:55:31.977Z	INFO	[monitoring]	log/log.go:118	Starting metrics logging every 30s
2020-02-09T00:55:31.977Z	INFO	instance/beat.go:429	filebeat start running.
2020-02-09T00:55:31.978Z	INFO	registrar/registrar.go:145	Loading registrar data from /usr/share/filebeat/data/registry/filebeat/data.json
2020-02-09T00:55:31.979Z	INFO	registrar/registrar.go:152	States Loaded from registrar: 12
2020-02-09T00:55:31.979Z	WARN	beater/filebeat.go:368	Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-02-09T00:55:31.979Z	INFO	crawler/crawler.go:72	Loading Inputs: 0
2020-02-09T00:55:31.979Z	INFO	crawler/crawler.go:106	Loading and starting Inputs completed. Enabled inputs: 0
2020-02-09T00:55:31.979Z	WARN	[cfgwarn]	kubernetes/kubernetes.go:59	BETA: The kubernetes autodiscover is beta
2020-02-09T00:55:31.984Z	INFO	kubernetes/util.go:94	kubernetes: Using pod name filebeat-filebeat-bl6x6 and namespace default to discover kubernetes node
2020-02-09T00:55:32.042Z	INFO	kubernetes/util.go:100	kubernetes: Using node kind-worker discovered by in cluster pod node query
2020-02-09T00:55:32.043Z	INFO	[autodiscover]	autodiscover/autodiscover.go:104	Starting autodiscover manager
2020-02-09T00:55:32.147Z	INFO	log/input.go:152	Configured paths: [/var/lib/docker/containers/49ebfad7ad3775ba66e6e775bc089fa27b79fe59efa016e695791f6c799e9945/*-json.log]
2020-02-09T00:55:32.148Z	INFO	input/input.go:114	Starting input of type: container; ID: 9137770389487786806 
2020-02-09T00:55:32.149Z	INFO	log/input.go:152	Configured paths: [/var/lib/docker/containers/406c310c7e514bd2e603105c1946a5944cc8f7695923997a38523897415e788d/*-json.log]
2020-02-09T00:55:32.150Z	INFO	input/input.go:114	Starting input of type: container; ID: 15042780179735339830 
2020-02-09T00:55:32.151Z	INFO	log/input.go:152	Configured paths: [/var/lib/docker/containers/5dfb2f8cf7478e9ae993ffa6d182eaff4e8ed3d8e24094b0233ee3711981befb/*-json.log]
2020-02-09T00:55:32.151Z	INFO	input/input.go:114	Starting input of type: container; ID: 11430920626231009588 
2020-02-09T00:55:32.154Z	INFO	log/input.go:152	Configured paths: [/var/lib/docker/containers/fb0fed987f7ac758030deab2fa09e96359c7e51dd4df0695fe6213dc2af90f45/*-json.log]
2020-02-09T00:55:32.154Z	INFO	input/input.go:114	Starting input of type: container; ID: 15751043935070790471 
2020-02-09T00:55:32.156Z	INFO	log/input.go:152	Configured paths: [/var/lib/docker/containers/e95db206a1050d819a20f9c5509ff158309a4b00ce2d0a34d5aea9271389bbb6/*-json.log]
2020-02-09T00:55:32.156Z	INFO	input/input.go:114	Starting input of type: container; ID: 9433795024465231413 
2020-02-09T00:55:32.156Z	INFO	log/input.go:152	Configured paths: [/var/lib/docker/containers/a52f0d70b9db6f884e96ff57f66b005422f90139d6dbf61b36073a71aee4263b/*-json.log]
2020-02-09T00:55:32.156Z	INFO	input/input.go:114	Starting input of type: container; ID: 18412748010883190238 
2020-02-09T00:55:32.158Z	INFO	log/input.go:152	Configured paths: [/var/lib/docker/containers/2b51132d5c1a1bc42395cad57778cc1ebe62b97c7db526ba8ead29c3f08c685d/*-json.log]
2020-02-09T00:55:32.158Z	INFO	input/input.go:114	Starting input of type: container; ID: 13976086098406123311 
2020-02-09T00:55:32.429Z	INFO	log/input.go:152	Configured paths: [/var/lib/docker/containers/dbdd578dbe58089c32bfaf78c0c32619f46f48646e67eb6276d10fcadc456490/*-json.log]
2020-02-09T00:55:32.429Z	INFO	input/input.go:114	Starting input of type: container; ID: 14861032766883028300 
2020-02-09T00:55:43.985Z	INFO	input/input.go:149	input ticker stopped
2020-02-09T00:55:43.985Z	INFO	input/input.go:167	Stopping Input: 14861032766883028300
2020-02-09T00:55:43.986Z	INFO	log/input.go:152	Configured paths: [/var/lib/docker/containers/dbdd578dbe58089c32bfaf78c0c32619f46f48646e67eb6276d10fcadc456490/*-json.log]
2020-02-09T00:55:43.986Z	INFO	input/input.go:114	Starting input of type: container; ID: 14861032766883028300 
2020-02-09T00:56:01.947Z	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":140,"time":{"ms":147}},"total":{"ticks":330,"time":{"ms":340},"value":330},"user":{"ticks":190,"time":{"ms":193}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":8},"info":{"ephemeral_id":"eccb85cf-2921-47c4-bda2-bf6c0aee80c0","uptime":{"ms":30111}},"memstats":{"gc_next":10350064,"memory_alloc":5516560,"memory_total":15688664,"rss":49332224},"runtime":{"goroutines":75}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"kafka"},"pipeline":{"clients":8,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":3},"load":{"1":5.82,"15":13.8,"5":9.63,"norm":{"1":1.94,"15":4.6,"5":3.21}}}}}}
2020-02-09T00:56:04.583Z	INFO	beater/filebeat.go:443	Stopping filebeat
2020-02-09T00:56:04.583Z	INFO	[autodiscover]	cfgfile/list.go:118	Stopping 8 runners ...
2020-02-09T00:56:04.583Z	INFO	input/input.go:149	input ticker stopped
2020-02-09T00:56:04.584Z	INFO	input/input.go:167	Stopping Input: 15042780179735339830
2020-02-09T00:56:04.584Z	INFO	input/input.go:149	input ticker stopped
2020-02-09T00:56:04.584Z	INFO	input/input.go:167	Stopping Input: 11430920626231009588
2020-02-09T00:56:04.584Z	INFO	input/input.go:149	input ticker stopped
2020-02-09T00:56:04.584Z	INFO	input/input.go:167	Stopping Input: 15751043935070790471
2020-02-09T00:56:04.584Z	INFO	input/input.go:149	input ticker stopped
2020-02-09T00:56:04.584Z	INFO	input/input.go:167	Stopping Input: 9433795024465231413
2020-02-09T00:56:04.584Z	INFO	input/input.go:149	input ticker stopped
2020-02-09T00:56:04.584Z	INFO	input/input.go:167	Stopping Input: 18412748010883190238
2020-02-09T00:56:04.584Z	INFO	input/input.go:149	input ticker stopped
2020-02-09T00:56:04.584Z	INFO	input/input.go:167	Stopping Input: 13976086098406123311
2020-02-09T00:56:04.584Z	INFO	input/input.go:149	input ticker stopped
2020-02-09T00:56:04.584Z	INFO	input/input.go:167	Stopping Input: 14861032766883028300
2020-02-09T00:56:04.584Z	INFO	input/input.go:149	input ticker stopped
2020-02-09T00:56:04.584Z	INFO	input/input.go:167	Stopping Input: 9137770389487786806
2020-02-09T00:56:04.584Z	INFO	[autodiscover]	autodiscover/autodiscover.go:292	Stopped autodiscover manager
2020-02-09T00:56:04.584Z	INFO	crawler/crawler.go:139	Stopping Crawler
2020-02-09T00:56:04.584Z	INFO	crawler/crawler.go:149	Stopping 0 inputs
2020-02-09T00:56:04.585Z	INFO	crawler/crawler.go:165	Crawler stopped
2020-02-09T00:56:04.585Z	INFO	registrar/registrar.go:367	Stopping Registrar
2020-02-09T00:56:04.585Z	INFO	registrar/registrar.go:293	Ending Registrar
2020-02-09T00:56:04.597Z	INFO	[monitoring]	log/log.go:153	Total non-zero metrics	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":150,"time":{"ms":157}},"total":{"ticks":340,"time":{"ms":355},"value":340},"user":{"ticks":190,"time":{"ms":198}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":8},"info":{"ephemeral_id":"eccb85cf-2921-47c4-bda2-bf6c0aee80c0","uptime":{"ms":32763}},"memstats":{"gc_next":10350064,"memory_alloc":5753744,"memory_total":15925848,"rss":49332224},"runtime":{"goroutines":13}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"kafka"},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":12},"writes":{"success":1,"total":1}},"system":{"cpu":{"cores":3},"load":{"1":5.82,"15":13.8,"5":9.63,"norm":{"1":1.94,"15":4.6,"5":3.21}}}}}}
2020-02-09T00:56:04.599Z	INFO	[monitoring]	log/log.go:154	Uptime: 32.7656174s
2020-02-09T00:56:04.600Z	INFO	[monitoring]	log/log.go:131	Stopping metrics logging.
2020-02-09T00:56:04.600Z	INFO	instance/beat.go:435	filebeat stopped.

Here is my filebeat confg file

filebeat.autodiscover:
      providers:
        - type: kubernetes
          hints.enabled: true
      processors:
        - add_kubernetes_metadata:
            in_cluster: true
output.kafka:
      codec.format:
        string: '%{[@timestamp]} %{[message]}'
      hosts: ["kafka:9092"]
      topic: 'log'
      partition.round_robin:
        reachable_only: false
      required_acks: 1
      compression: gzip
      max_message_bytes: 1000000
logging.level: info

faec · February 10, 2020, 8:29pm

Welcome! Could you say a little more about this? The only obvious problems in that output are the warnings about pipeline configuration (which makes sense since you're outputting to Kafka rather than Elasticsearch). What does "crashloopbackoff" refer to? Is Filebeat shutting down when you don't expect it to? If you give more details about your setup and how Filebeat's behavior is unexpected, we can help more with troubleshooting it.

izazahamed.babji · February 13, 2020, 1:58am

I tried removing the liveness probe and its working fine now! Now, I am stuck in a different issue!

filebeat.inputs:
    - type: container
      paths:
        - /var/log/containers/audit-.*\.log
      exclude_files:
        - filebeat-.*\.log
        - logstash-.*\.log
        - kafka-.*\.log
        - kibana-.*\.log
        - kindnet-*\.log
    processors:
      - add_kubernetes_metadata:
          host: ${NODE_NAME}
          in_cluster: true
          default_matchers.enabled: false
          matchers:
          - logs_path:
              logs_path: /var/log/containers/
      - add_cloud_metadata:
      - add_host_metadata:
    output.kafka:
      hosts: ["kafka-headless:9092"]
      topic: db
      partition.round_robin:
        reachable_only: false
      required_acks: 1

I only want the logs from audit app but I am not receiving any logs!
Also I checked the filebeat container in the node where the audit pod is present... there is an entry of audit in the filebeat/data/registry/filebeat/data.json

{"source":"/var/log/containers/audit-db-78586c6dc5-qbs65_default_audit-db-b342ddc17cfbe6b9c99ac6852f5e070eb58f716e39e23255554fd7f831972ba7.log","offset":12040,"timestamp":"2020-02-13T01:24:57.9412508Z","ttl":-2,"type":"container","meta":null,"FileStateOS":{"inode":4329827,"device":2049}}

What am I missing?

system · March 12, 2020, 2:13am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.