Elastic 8.11.3 on docker in OSX silicon has disk volume sizing issues

[matthal (Matt)]

Trying to run elasticsearch locally for development using docker compose (Getting started with the Elastic Stack and Docker-Compose | Elastic Blog)

I end up getting disk pressure issues, well a warning, but it ends up meaning a dysfunctional es when you try adding indexes and data.

elasticsearch-es01-1   | {"@timestamp":"2023-12-17T21:46:42.285Z", "log.level": "WARN", "message":"high disk watermark [90%] exceeded on [8QrPMNt3TZ2KKJrckbbXOg][es01][/usr/share/elasticsearch/data] free: 14.9gb[6.4%], shards will be relocated away from this node; currently relocating away shards totalling [0] bytes; the node is expected to continue to exceed the high disk watermark when these relocations are complete", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.routing.allocation.DiskThresholdMonitor","elasticsearch.cluster.uuid":"Xjc0-pxlSyqehu4SKDBvfw","elasticsearch.node.id":"8QrPMNt3TZ2KKJrckbbXOg","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}

To stop this happening you can configure the following, but that just avoids the real issue.

curl -XPUT "http://localhost:9200/_cluster/settings" \
 -H 'Content-Type: application/json' -d'
{
  "persistent": {
    "cluster": {
      "routing": {
        "allocation.disk.threshold_enabled": false
      }
    }
  }
}'

The real issue is that the docker image thinks it has used up all the space, whereas docker demon correctly says what it is actually using.

docker ps -s
CONTAINER ID   IMAGE                                                  COMMAND                  CREATED         STATUS                   PORTS                              NAMES                   SIZE
8b4f21221f4f   docker.elastic.co/elasticsearch/elasticsearch:8.11.3   "/bin/tini -- /usr/l…"   8 minutes ago   Up 8 minutes (healthy)   0.0.0.0:9200->9200/tcp, 9300/tcp   elasticsearch-es01-1    246kB (virtual 797MB)
0fe87d614464   docker.elastic.co/elasticsearch/elasticsearch:8.11.3   "/bin/tini -- /usr/l…"   8 minutes ago   Up 8 minutes (healthy)   9200/tcp, 9300/tcp                 elasticsearch-setup-1   688MB (virtual 1.49GB)

Looks normal, but lets look inside the running container

docker exec -it 8b4f21221f4f /bin/bash
elasticsearch@8b4f21221f4f:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay         234G  207G   15G  94% /
tmpfs            64M     0   64M   0% /dev
shm              64M     0   64M   0% /dev/shm
/dev/vda1       234G  207G   15G  94% /etc/hosts
tmpfs            20G     0   20G   0% /sys/firmware

Has anyone else experienced this scenario?

[stephenb (Stephen Brown)]

Hi @matthal Welcome to the commuity.

@matthal

/dev/vda1 234G 207G 15G 94% /etc/hosts

What do you have the Virtual Disk Limit Set to?

That is what is being reflected if you run

GET /_cat/nodes/?v&h=name,du,dt,dup,hp,hc,rm,rp,r

or

sh-5.0$ df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay          59G   37G   19G  66% /
tmpfs            64M     0   64M   0% /dev
shm              64M     0   64M   0% /dev/shm
/dev/vda1        59G   37G   19G  66% /etc/hosts <!---- Here 
tmpfs           3.9G     0  3.9G   0% /proc/acpi
tmpfs           3.9G     0  3.9G   0% /sys/firmware

Note: mine is set to 64GB or as the note says
"Due to filesystem overhead, the real available space might be less."

So your data is still within the Docker Virtual Disk... It is just persistent...

This is docker disk stuff...

So you need to clean up something else that is taking up space on your Docker Virtual disk or increase it.

You can read more about docker storage here...

Manage data in Docker

[matthal (Matt)]

I have the following

curl -XGET 'http://localhost:9200/_cat/nodes/?v&h=name,du,dt,dup,hp,hc,rm,rp,r'

name      du      dt   dup hp      hc  rm rp r
es01 218.6gb 233.6gb 93.59 55 283.6mb 1gb 99 cdfhilmrstw

df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay         234G  207G   15G  94% /
tmpfs            64M     0   64M   0% /dev
shm              64M     0   64M   0% /dev/shm
/dev/vda1       234G  207G   15G  94% /etc/hosts
tmpfs            20G     0   20G   0% /sys/firmware

I am not sure how Elasticsearch is using 207 G when there is no data in this, i.e. it is a fresh install.

[matthal (Matt)]

Here is a snapshot of the volume in docker desktop too

Thank you for your help @stephenb

[leandrojmp (Leandro Pereira)]

@matthal

I am not sure how Elasticsearch is using 207 G when there is no data in this, i.e. it is a fresh install.

Do you have any other docker containers? This is space will be used by everything you run on Docker on your Mac, it is not Elasticsearch that is using it.

This is related to Docker, not Elasticsearch.

[stephenb (Stephen Brown)]

@matthal

as @leandrojmp said, that is the share docker virtual disk

You can run

docker system df

and for the details...

docker system df -v

to see what else is taking up space

[matthal (Matt)]

@stephenb

docker system df -v

Thanks for the reply. This makes sense.

docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          27        4         45.1GB    42.88GB (95%)
Containers      5         5         688.9MB   0B (0%)
Local Volumes   21        3         101.8GB   101.8GB (99%)
Build Cache     149       0         19.36GB   19.36GB

I removed some of the other volumes used by containers not running and got down to the following (there are some I can't remove as they are needed)

$ docker system df      
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          27        4         45.1GB    42.88GB (95%)
Containers      5         5         688.9MB   0B (0%)
Local Volumes   10        3         48.46GB   48.46GB (99%)
Build Cache     149       0         19.36GB   19.36GB

This reduction is reflected in the following:

df -h (in elastic container)
Filesystem      Size  Used Avail Use% Mounted on
overlay         234G  156G   67G  70% /
tmpfs            64M     0   64M   0% /dev
shm              64M     0   64M   0% /dev/shm
/dev/vda1       234G  156G   67G  70% /etc/hosts
tmpfs            20G     0   20G   0% /sys/firmware

Which then changed the health status of the cluster from [RED] to [GREEN].

I'll try to figure out why there is another 30 GB on top of the 48.46 GB reported that df -h inside the container is seeing.

In any case, now I understand what is happening here, thank you.

regards
Matt