Thanks @pierhugues for your reply. I have enabled debug level for "add_docker_metadata" selector and got one one output line repeated >thousands of times like this:
{"log":"2019-03-06T15:33:41.845Z\u0009DEBUG\u0009[add_docker_metadata]\u0009add_docker_metadata/add_docker_metadata.go:186\u0009Container not found: cid=badb87a5b17a6493a0deacdd086aae713c904b3a2d511b9996e64f01b458b300-json.log\n","stream":"stderr","attrs":{"tag":"docker/elastic-filebeat"},"time":"2019-03-06T15:33:41.845724496Z"}
This line is repeated in the output maybe 100 times per second so I think there is something strange going on. I will try to get to debug log files again and will disable the log rotation to get the beginning logs.
So long I could only figure out the place where those log lines were produced. Seems like the last line of this:
container := d.watcher.Container(cid)
if container != nil {
meta := common.MapStr{}
metaIface, ok := event.Fields["docker"]
if ok {
meta = metaIface.(common.MapStr)
}
if len(container.Labels) > 0 {
labels := common.MapStr{}
for k, v := range container.Labels {
safemapstr.Put(labels, k, v)
}
meta.Put("container.labels", labels)
}
meta.Put("container.id", container.ID)
meta.Put("container.image", container.Image)
meta.Put("container.name", container.Name)
event.Fields["docker"] = meta.Clone()
} else {
d.log.Debugf("Container not found: cid=%s", cid)
}