Filebeat daemonset randomly deletes itself?!


(Karl Stoney) #1

Hey,
I'm trying to run the filebeat docker container on my kubernetes cluster.
It seems to come up... but then randomly, after about 30 seconds it disapears, and the daemonset, and associated configmaps are gone!?

Logs from a pod, as you can see it stops, and then the daemonset is gone. When i run delete, it appears the configmaps are gone too?!

2018/01/30 10:34:06.631577 metrics.go:51: INFO Total non-zero values:  beat.memstats.gc_next=20003056 beat.memstats.memory_alloc=10290496 beat.memstats.memory_total=36014432 filebeat.events.active=3313 filebeat.events.added=3318 filebeat.events.done=5 filebeat.harvester.open_files=10 filebeat.harvester.running=10 filebeat.harvester.started=10 libbeat.config.module.running=1 libbeat.config.module.starts=1 libbeat.config.reloads=2 libbeat.output.type=elasticsearch libbeat.pipeline.clients=0 libbeat.pipeline.events.active=3308 libbeat.pipeline.events.filtered=10 libbeat.pipeline.events.published=3308 libbeat.pipeline.events.total=3318 registrar.states.current=5 registrar.states.update=5 registrar.writes=7
2018/01/30 10:34:06.631659 metrics.go:52: INFO Uptime: 42.286487018s
2018/01/30 10:34:06.631667 beat.go:268: INFO filebeat stopped.                                                                                                         

❯ kks get daemonset
No resources found.

❯ k delete -f filebeat-kubernetes.yaml
clusterrolebinding "filebeat" deleted
clusterrole "filebeat" deleted
serviceaccount "filebeat" deleted
Error from server (NotFound): error when deleting "filebeat-kubernetes.yaml": configmaps "filebeat-config" not found
Error from server (NotFound): error when deleting "filebeat-kubernetes.yaml": configmaps "filebeat-prospectors" not found
Error from server (NotFound): error when stopping "filebeat-kubernetes.yaml": daemonsets.extensions "filebeat" not found

(Carlos Pérez Aradros) #2

Hi @Karl_Stoney,

Filebeat has no code to delete itself from Kubernetes, I'm inclined to think that something/someone else is deleting it?

You can watch kubectl describe -n kube-system ds/filebeat to see the events affecting the daemonset, you can also consider checking audit logs

Best regards


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.