Monitoring docker logs, container can no longer be removed

Hi,

I've read several blog posts and dicussions on this forum for collecting the docker json logs using filebeat. I've come up with the following config:

filebeat.prospectors:
- input_type: log
 paths:
 - /var/lib/docker/containers/*/*.log
 document_type: docker
 json.message_key: log
 json.keys_under_root: true
 processors:
 - add_docker_metadata: ~

Which works great except for one nasty side effect, this causes docker (or docker-compose) to not be-able to remove or recreate docker containers. We're getting the following errors when trying to recreate containers:

/var/lib/docker/containers/096ea09c9772a40b2c398e2ee25a172a16f635b7cc8c4141d2068212d2041c16/shm: device or resource busy

We've investigated what processes where using that file, and the it was caused by Filebeat. I couldn't find anyone with the same problems, so I hope i'm doing something very simple wrong. Any ideas would be greatly appreciated.

This is something I haven't experienced, in theory filebeat should not open that file at all. What version of filebeat are you using? Could you share the full filebeat.yml?

Best regards

We are using filebeat inside a docker container:

FROM docker.elastic.co/beats/filebeat:6.0.0

We've mounted the folders like:

volumes:
 - "/var/lib/docker/containers/:/var/lib/docker/containers/:ro"
 - "/var/run/docker.sock:/var/run/docker.sock:ro"

The only section missing from the orignal post is the output:

output.logstash:
 hosts: ["logstash:5044"]

I see, could you paste the result of running lsof on the blocked shm file?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.