Watcher.go logs old docker container events as if they are new and may log error if container has been removed

Elastic Stack Beats
,
1

We are using filebeat to publish our application logs to graylog. Both our application and filebeat are running in docker containers.

When we turn on debug logging in filebeat, we observe that a debug logging statement in watcher.go reprints "Got a new docker event" for the last event it has received about every 10 minutes. For example, we may see log statements like this:

{"log.level":"debug","@timestamp":"2022-07-11T18:45:24.821Z","log.logger":"add_docker_metadata","log.origin":{"file.name":"docker/watcher.go","file.line":280},"message":"Got a new docker event: {destroy 0d48b5ffbcae2c44cc9ed694ab383b43554629b747eb6bc8b7261c6300ebf7e9 mongo:4.2.10 container destroy {0d48b5ffbcae2c44cc9ed694ab383b43554629b747eb6bc8b7261c6300ebf7e9 map[image:mongo:4.2.10 name:musing_herschel]} local 1657564516 1657564516448793660}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-07-11T18:55:34.821Z","log.logger":"add_docker_metadata","log.origin":{"file.name":"docker/watcher.go","file.line":280},"message":"Got a new docker event: {destroy 0d48b5ffbcae2c44cc9ed694ab383b43554629b747eb6bc8b7261c6300ebf7e9 mongo:4.2.10 container destroy {0d48b5ffbcae2c44cc9ed694ab383b43554629b747eb6bc8b7261c6300ebf7e9 map[image:mongo:4.2.10 name:musing_herschel]} local 1657564516 1657564516448793660}","service.name":"filebeat","ecs.version":"1.6.0"}

In this example, the last docker event that was received was a container destroy event, and it was relogged as if it had just been received. This seems not to be the intent of the watcher.

In addition, we see that in some cases, for example if a container start event is reprocessed after that container that generated it is destroyed, watcher.go logs an error level log like this:

ERROR docker/watcher.go:274 Error getting container info: <nil>

I have seen the error level log in filebeat 7.6.0 while I haven't (yet) reproduced the problem in filebeat 8.3.2, I have seen that docker events are logged multiple times as shown in my example above, suggesting that the root cause has not been addressed in the current latest version. Is this a known bug? If not, I would like to ensure it has been properly reported.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.