Hello Team
Problem
We have an issue, where logfile from a restarted docker container is not scanned, registered and processed by a filebeat.
We were using docker input module without problems, until we have spotted a couple of missing log events.
Detailed description:
On all of our virtuals we have installed filebeat service for shipping logs to Logstash and Elasticsearch. On machines where we run docker we have configured docker input module to ship docker logs.
One of our services is running in a docker container on 3 virtuals and we have restarted it (with new parameters) in roughly same time, which has generated loglines into a new docker files.
We spotted that only one of these three logs was shipped. Later filebeat service on the second machine was restarted and missing logs from this machine was shipped too.
We examined the third machine, from which no logs were shipped yet and found out, that new docker log was not written to a registry file. Also there isn't any mention in the filebeat log, that the new file is processed (with loglevel: info). Other docker logs from this filebeat instance are shipped without problems.
We consider this is a bug, which is quite serious, because it silently ignores information from a newly started docker containers.
Our filebeat configuration for docker input module:
filebeat.inputs:
- type: docker
enabled: true
ignore_older: 24h
close_inactive: 1m
clean_inactive: 48h
clean_removed: true
containers.ids: '*'
combine_partial: true
cri.parse_flags: true
processors:
- add_docker_metadata:
labels.dedot: true
fields:
service: docker
loglevel: DEBUG
fields_under_root: true
Any help with this issue would be very appreciated.