Logfile from restarted docker container is not scanned and written to registry

Elastic Stack Beats
,
1

Hello Team

Problem
We have an issue, where logfile from a restarted docker container is not scanned, registered and processed by a filebeat.
We were using docker input module without problems, until we have spotted a couple of missing log events.

Detailed description:
On all of our virtuals we have installed filebeat service for shipping logs to Logstash and Elasticsearch. On machines where we run docker we have configured docker input module to ship docker logs.

One of our services is running in a docker container on 3 virtuals and we have restarted it (with new parameters) in roughly same time, which has generated loglines into a new docker files.
We spotted that only one of these three logs was shipped. Later filebeat service on the second machine was restarted and missing logs from this machine was shipped too.

We examined the third machine, from which no logs were shipped yet and found out, that new docker log was not written to a registry file. Also there isn't any mention in the filebeat log, that the new file is processed (with loglevel: info). Other docker logs from this filebeat instance are shipped without problems.

We consider this is a bug, which is quite serious, because it silently ignores information from a newly started docker containers.

Our filebeat configuration for docker input module:

    filebeat.inputs:
    - type: docker
      enabled: true
      ignore_older: 24h
      close_inactive: 1m
      clean_inactive: 48h
      clean_removed: true
      containers.ids: '*'
      combine_partial: true
      cri.parse_flags: true
      processors:
        - add_docker_metadata:
            labels.dedot: true
      fields:
        service: docker
        loglevel: DEBUG
      fields_under_root: true

Any help with this issue would be very appreciated.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

4

Hey @Tomas_Bartek,

I think this can be an issue already solved in 7.x versions. I see in the issue you have opened in Github that you are using 6.8. Would you have the chance to try with a 7.x version?

Maybe this change fixed this issue: https://github.com/elastic/beats/issues/12310, it was released in 7.3.

If you cannot upgrade to 7.x, maybe you can try to compile filebeat with the mentioned patch.

If this fixes your issue we might consider backporting the fix to 6.8.

Thanks for reporting!