Elasticsearch Docker Container filling up disk

Elastic Stack Elasticsearch
1

I've been doing some experimentation and development with Elasticsearch and Kibana running in the official docker containers. I've got the ElasticSearch /usr/share/elasticsearch/data mapped to a 1.5TB drive which seems to be automatically maintaining the disk space to 95%.

However, for some reason my / drive (which also handles the /var/lib/docker/overlay2/ container storage) has filled up.

How can I figure out what inside the docker containers (either Kibana or Elasticsearch) is filling up my / drive? Should I relocate my default /var/lib/docker to the external /data drive?

My docker-compose configs are like this:

- name: Build Elasticsearch Docker Image
  docker_service:
    project_name: elasticsearch
    definition:
      version: '3'
      services:
        elasticsearch:
          image: docker.elastic.co/elasticsearch/elasticsearch:{{ elastic_version }}
          container_name: elasticsearch
          restart_policy:
            condition: on-failure
            delay: 5s
            max_attempts: 3
            window: 120s
          environment:
            - bootstrap.memory_lock=true
            - "ELASTIC_PASSWORD=changeme"
            - "ES_JAVA_OPTS=-Xms16g -Xmx16g"
            - xpack.security.enabled=false
          ulimits:
            memlock:
              soft: -1
              hard: -1
            nofile:
              soft: 65536
              hard: 65536
          volumes:
            - /data/elasticsearch/data:/usr/share/elasticsearch/data
            - /var/log/elasticsearch:/var/log/elasticsearch
          ports:
            - 9200:9200
          networks: ['stack']

      networks: {stack: {}}
 
- name: Build Kibana Docker Image
  docker_service:
    project_name: kibana
    definition:
      version: '3'
      services:
        kibana:
          image: docker.elastic.co/kibana/kibana:{{ kibana_version }}
          container_name: kibana
          restart_policy:
            condition: on-failure
            delay: 5s
            max_attempts: 3
            window: 120s
          environment:
            - ELASTICSEARCH_URL=http://localhost:9200
          ports:
            - 5601:5601
          networks: ['stack']

      networks: {stack: {}}
2

How much space do you have on /root?

Are you able to send a screenshot after running the following command:

df -h

1 Like
3

Are copious logs being written to /? A healthy Elasticsearch cluster writes very few logs, so this might indicate a problem that needs addressing.

1 Like
4

In the end, I couldn't even shell into the docker container due to disk full. I ended up rm'ing the docker containers, moving the /var/lib/docker directory to my larger "/data" drive, and doing another docker-compose to pull the new images. Hopefully this works ok with the extra storage headroom.

sudo df -h
Filesystem                           Size  Used Avail Use% Mounted on
/dev/mapper/elk--alln--001--vg-root   39G   39G     0 100% /
/dev/sda1                            472M  132M  316M  30% /boot
/dev/mapper/data-data1               1.5T  1.5T   79G  95% /data
overlay                               39G   39G     0 100% /var/lib/docker/overlay2/1b9b77164ac19e291491b12151c9fe26a1551d72b3c66c015b43bcd1ff418770/merged
5

I linked /var/lib/docker to /data/docker (a bigger drive) then re-pulled the elasticsearch and kibana docker images. Hopefully this helps not kill my / drive next time.

1 Like
6

This will work, i usually have the same problem so i usually assign 200gb (Running 10+ docker images) for my /var partition or just change the default docker directory

1 Like
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.