We have an issue, where logfile from a restarted docker container is not scanned, registered and processed by a filebeat.
We were using docker input module without problems, until we have spotted a couple of missing log events.
On all of our virtuals we have installed filebeat service for shipping logs to Logstash and Elasticsearch. On machines where we run docker we have configured docker input module to ship docker logs.
One of our services is running in a docker container on 3 virtuals and we have restarted it (with new parameters) in roughly same time, which has generated loglines into a new docker files.
We spotted that only one of these three logs was shipped. Later filebeat service on the second machine was restarted and missing logs from this machine was shipped too.
We examined the third machine, from which no logs were shipped yet and found out, that new docker log was not written to a registry file. Also there isn't any mention in the filebeat log, that the new file is processed (with loglevel: info). Other docker logs from this filebeat instance are shipped without problems.
We consider this is a bug, which is quite serious, because it silently ignores information from a newly started docker containers.
Our filebeat configuration for docker input module:
filebeat.inputs: - type: docker enabled: true ignore_older: 24h close_inactive: 1m clean_inactive: 48h clean_removed: true containers.ids: '*' combine_partial: true cri.parse_flags: true processors: - add_docker_metadata: labels.dedot: true fields: service: docker loglevel: DEBUG fields_under_root: true
Any help with this issue would be very appreciated.