I have the elastic stack (elasticsearch,logstash,kibana) + filebeat + metricbeat running on a 14.04.5 LTS (GNU/Linux 3.13.0-115-generic x86_64) host.
All elastic software is running in their own Docker container. So I have 5 containers for the above mentioned pieces.
When trying to update one of the containers (using docker-compose) I get a complaint that the internal container directory can't be removed.
For example I try to update kibana which means removing the kibana container and then recreate it:
docker-compose up -d kibana
elk_elasticsearch_1 is up-to-date
Recreating elk_kibana_1
ERROR: for kibana Unable to remove filesystem for
77e6efe9cf26c25af63037be3f77013e49a529e4ae6fe93ef75b1b246124c546: remove /var/lib/docker/containers/77e6efe9cf26c25af63037be3f77013e49a529e4ae6fe93ef75b1b246124c546/shm: device or resource busy
ERROR: Encountered errors while bringing up the project.
The following command will tell me that the container id is associated with the container for kibana
docker ps -a | grep 77e6e
After I first suspected filebeat to be responsible for locking the container directory (as it mounts the Docker host directory /var/lib/docker/containers into the filebeat container). But stopping filebeat didn't resolve the issue.
So, next I stopped metricbeat to experience that this would allow the container to be deleted:
docker-compose stop metricbeat
Stopping elk_metricbeat_1 ... done
docker-compose rm kibana
Going to remove elk_kibana_1
Are you sure? [yN] y
Removing elk_kibana_1 ... done
Here is my metricbeat configuration:
#========================== Modules configuration ============================
metricbeat.modules:
#------------------------------- Docker Module -------------------------------
- module: docker
metricsets: ["container", "cpu", "diskio", "healthcheck", "info", "memory", "network"]
hosts: ["unix:///var/run/docker.sock"]
enabled: true
period: 10s
#-------------------------- Elasticsearch output -------------------------------
output.elasticsearch:
# Boolean flag to enable or disable the output module.
enabled: true
hosts: ["elasticsearch:9200"]
username: "${ELASTICSEARCH_USERNAME:elastic}"
password: "${ELASTICSEARCH_PASSWORD:changeme}"
The following is the docker-compose configuration:
version: "2.1"
services:
metricbeat:
build: ./metricbeat
group_add: ['root', 'adm']
user: root
volumes:
- /proc:/hostfs/proc:ro
- /sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro
- /:/hostfs:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
command: metricbeat -e -system.hostfs=/hostfs
depends_on: {elasticsearch: {condition: service_healthy}}
stop_grace_period: 1m0s
restart: always
I have also tried that on my macOS (Sierra) with Docker for Mac Version 17.06.0-ce-mac18 (18433). Here the issue does not show up. So it might something related to the OS platform - just guessing?
