Filebeat stopped harvesting logs after a while

We have filebeat 6.7.1 running as a daemonset on our kubernetes cluster. Everything was working fine till suddenly logs stopped popping up in ES.

When we checked filebeat logs, there was no ERROR but as filebeat there was nothing to harvest:

2019-07-31T12:30:08.604Z    INFO    [monitoring]    log/log.go:144    Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":35080},"total":{"ticks":229250,"value":229250},"user":{"ticks":194170}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":5},"info":{"ephemeral_id":"91360383-6212-4667-90be-daef7d298991","uptime":{"ms":2579430011}},"memstats":{"gc_next":4194304,"memory_alloc":2155264,"memory_total":22969148448}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.51,"15":0.26,"5":0.34,"norm":{"1":0.1275,"15":0.065,"5":0.085}}}}}} 

But no sooner we restarted the pod, logs started getting harvested and we could now see the logs in es.

Is this a known bug in the version we're using? How do we debug this further?

Hmm might be a good idea to share your filebeat.yml config just for reference.

Maybe take a look at autodiscover, specifically the kubernetes section: https://www.elastic.co/guide/en/beats/filebeat/6.7/configuration-autodiscover.html#_kubernetes