# Monitoring docker logs, container can no longer be removed

**URL:** https://discuss.elastic.co/t/monitoring-docker-logs-container-can-no-longer-be-removed/111494
**Category:** Beats
**Tags:** filebeat
**Created:** [December 13, 2017, 7:08am UTC](https://discuss.elastic.co/t/monitoring-docker-logs-container-can-no-longer-be-removed/111494 "2017-12-13T07:08:24Z")
**Posts on this page:** 5
**Page:** 1

<div class="post-metadata">

### Author: ![Jordi\_Jansen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jordi_jansen/32/25483_2.png) [@Jordi\_Jansen](https://discuss.elastic.co/u/Jordi_Jansen)
#### Post date: [December 13, 2017, 7:08am UTC](https://discuss.elastic.co/t/monitoring-docker-logs-container-can-no-longer-be-removed/111494/1 "2017-12-13T07:08:24Z")

</div>

Hi,

I've read several blog posts and dicussions on this forum for collecting the docker json logs using filebeat. I've come up with the following config:

```auto
filebeat.prospectors:
- input_type: log
 paths:
 - /var/lib/docker/containers/*/*.log
 document_type: docker
 json.message_key: log
 json.keys_under_root: true
 processors:
 - add_docker_metadata: ~

```

Which works great except for one nasty side effect, this causes docker (or docker-compose) to not be-able to remove or recreate docker containers. We're getting the following errors when trying to recreate containers:

```auto
/var/lib/docker/containers/096ea09c9772a40b2c398e2ee25a172a16f635b7cc8c4141d2068212d2041c16/shm: device or resource busy

```

We've investigated what processes where using that file, and the it was caused by Filebeat. I couldn't find anyone with the same problems, so I hope i'm doing something very simple wrong. Any ideas would be greatly appreciated.

---

<div class="post-metadata">

### Author: ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)
#### Post date: [December 14, 2017, 11:23am UTC](https://discuss.elastic.co/t/monitoring-docker-logs-container-can-no-longer-be-removed/111494/2 "2017-12-14T11:23:43Z")

</div>

This is something I haven't experienced, in theory filebeat should not open that file at all. What version of filebeat are you using? Could you share the full filebeat.yml?

Best regards

---

<div class="post-metadata">

### Author: ![Jordi\_Jansen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jordi_jansen/32/25483_2.png) [@Jordi\_Jansen](https://discuss.elastic.co/u/Jordi_Jansen)
#### Post date: [December 14, 2017, 11:50am UTC](https://discuss.elastic.co/t/monitoring-docker-logs-container-can-no-longer-be-removed/111494/3 "2017-12-14T11:50:12Z")

</div>

We are using filebeat inside a docker container:

```auto
FROM docker.elastic.co/beats/filebeat:6.0.0

```

We've mounted the folders like:

```auto
volumes:
 - "/var/lib/docker/containers/:/var/lib/docker/containers/:ro"
 - "/var/run/docker.sock:/var/run/docker.sock:ro"

```

The only section missing from the orignal post is the output:

```auto
output.logstash:
 hosts: ["logstash:5044"]

```

---

<div class="post-metadata">

### Author: ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)
#### Post date: [December 14, 2017, 12:02pm UTC](https://discuss.elastic.co/t/monitoring-docker-logs-container-can-no-longer-be-removed/111494/4 "2017-12-14T12:02:36Z")

</div>

I see, could you paste the result of running `lsof` on the blocked `shm` file?

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [January 11, 2018, 12:02pm UTC](https://discuss.elastic.co/t/monitoring-docker-logs-container-can-no-longer-be-removed/111494/5 "2018-01-11T12:02:44Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
