Unable to bring containers down/stop containers whilst metricbeat is running

Elastic Stack Beats
,
1

Hi guys.

We have several hosts running metricbeat via docker and sometimes when bringing other containers down with docker-compose/stopping them also using docker-compose, we get the following error:

ERROR: for [container_name] driver "devicemapper" failed to remove root filesystem for [ID]: failed to remove device [Another ID]: Device is Busy

This is quite irritating as every time we have to bring metricbeat down, then bring the new container down then up again, and then metricbeat up again. I was wondering if there is something we can do to prevent it from happening?

Looked at a few different related posts around and it seems to be an issue not limited to me but there's been no listed resolutions, at least from what I can see.

Thanks,
Aaron

2

Wondering if anyone had a solution for this?

bump, i guess

3

Hi! Can you give us more information on how the containers are setup please?

4

Set up using docker-compose.

---
version: "2.1"
services:
    metricbeat:
        container_name: metricbeat
        hostname: redacted
        user: root
        image: docker.elastic.co/beats/metricbeat:${ELASTIC_VERSION}
        volumes:
          #Mount the metricbeat configuration so users can make edit
          - ./config/beats/metricbeat/metricbeat.yml:/usr/share/metricbeat/metricbeat.yml
          #Mount the modules.d directory into the container. This allows user to potentially make changes to the modules and they will be dynamically loaded.
          - ./config/beats/metricbeat/modules.d/:/usr/share/metricbeat/modules.d/
          # The commented sections below enable Metricbeat to monitor the Docker host rather than the Metricbeat container. These are used by the system module.
          - /proc:/hostfs/proc:ro
          - /sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro
          #Allows us to report on docker from the hosts information
          - /var/run/docker.sock:/var/run/docker.sock
          #We mount the host filesystem so we can report on disk usage with the system module
          - /:/hostfs:ro
          command: metricbeat -e -system.hostfs=/hostfs -E 
output.elasticsearch.hosts='["elasticsearch_url:9200"]' -E output.elasticsearch.username=elastic -E output.elasticsearch.password=${ES_PASSWORD} -strict.perms=false
        restart: on-failure
        network_mode: host
    networks:
      default:
        ipam:
          driver: redactedipam

I start it with docker-compose up -d metricbeat as there's other beats that we don't actually use (we could probably get rid of them).
I've got metricbeat.yml set up to auto-detect new modules in modules.d with the system and docker modules enabled. Docker seems to be the main issue, and modules.d/docker.yml is set up like so:

- module: docker
  metricsets: ["container", "cpu", "diskio", "healthcheck", "info", "memory", "network"]
  hosts: ["unix:///var/run/docker.sock"]
  enabled: true
  period: 60s

Do you need anything more?

Thanks

EDIT: we don't have Dockerfiles, in case you were wondering.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.