Monitoring docker logs, container can no longer be removed

[Jordi_Jansen]

Hi,

I've read several blog posts and dicussions on this forum for collecting the docker json logs using filebeat. I've come up with the following config:

filebeat.prospectors:
- input_type: log
 paths:
 - /var/lib/docker/containers/*/*.log
 document_type: docker
 json.message_key: log
 json.keys_under_root: true
 processors:
 - add_docker_metadata: ~

Which works great except for one nasty side effect, this causes docker (or docker-compose) to not be-able to remove or recreate docker containers. We're getting the following errors when trying to recreate containers:

/var/lib/docker/containers/096ea09c9772a40b2c398e2ee25a172a16f635b7cc8c4141d2068212d2041c16/shm: device or resource busy

We've investigated what processes where using that file, and the it was caused by Filebeat. I couldn't find anyone with the same problems, so I hope i'm doing something very simple wrong. Any ideas would be greatly appreciated.

[exekias (Carlos Pérez Aradros)]

This is something I haven't experienced, in theory filebeat should not open that file at all. What version of filebeat are you using? Could you share the full filebeat.yml?

Best regards

[Jordi_Jansen]

We are using filebeat inside a docker container:

FROM docker.elastic.co/beats/filebeat:6.0.0

We've mounted the folders like:

volumes:
 - "/var/lib/docker/containers/:/var/lib/docker/containers/:ro"
 - "/var/run/docker.sock:/var/run/docker.sock:ro"

The only section missing from the orignal post is the output:

output.logstash:
 hosts: ["logstash:5044"]

[exekias (Carlos Pérez Aradros)]

I see, could you paste the result of running lsof on the blocked shm file?