# Elasticsearch Docker: flood stage disk watermark \[95%\] exceeded

**URL:** <https://discuss.elastic.co/t/elasticsearch-docker-flood-stage-disk-watermark-95-exceeded/248479>\
**Category:** Elasticsearch\
**Tags:** docker\
**Created:** [September 14, 2020, 7:38am UTC](https://discuss.elastic.co/t/elasticsearch-docker-flood-stage-disk-watermark-95-exceeded/248479 "2020-09-14T07:38:33Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Stimmot](https://avatars.discourse-cdn.com/v4/letter/s/46a35a/32.png) [@Stimmot](https://discuss.elastic.co/u/Stimmot)\
**Post date:** [September 14, 2020, 7:38am UTC](https://discuss.elastic.co/t/elasticsearch-docker-flood-stage-disk-watermark-95-exceeded/248479/1 "2020-09-14T07:38:33Z")

</div>

I have a problem with my Elasticsearch nodes running in a docker environment. I'm starting them up with docker-compose and after a few minutes they tell me:  
_flood stage disk watermark [95%] exceeded_

I'm running it on a cluster with rather high storage capacity and I already tried to increase the watermark settings in the elasticsearch.yml file, but I still get the error. Maybe it has to do with the size of the docker containers.

Does anyone know what could be the problem? Any help is much appreciated.

The docker-compose.yml for reference:

```auto
version: '3.4'
services:
  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.8.1
    container_name: es01
    environment:
      #- discovery.type=single-node
      - node.name=es01
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es02,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - xpack.security.enabled=false
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data01:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - elastic

  es02:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.8.1
    container_name: es02
    environment:
      - node.name=es02
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data02:/usr/share/elasticsearch/data
    networks:
      - elastic

  es03:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.8.1
    container_name: es03
    environment:
      - node.name=es03
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es02
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data03:/usr/share/elasticsearch/data
    networks:
      - elastic

  kib01:
    image: docker.elastic.co/kibana/kibana:7.8.1
    container_name: kib01
    depends_on:
      - es01
      - es02
      - es03
    ports:
      - 5601:5601
    environment:
      ELASTICSEARCH_URL: http://es01:9200
      ELASTICSEARCH_HOSTS: http://es01:9200
    networks:
      - elastic

  client:
    image: appropriate/curl:latest
    depends_on:
      - es01
      - es02
      - es03
    networks:
      - elastic
    command: sh -c "curl es01:9200 && curl kib01:5601"

  dash_app:
    build: .
    ports:
    - 0.0.0.0:8050:8050
    depends_on:
      - es01
      - es02
      - es03
      - kib01
    networks:
      - elastic

#mapping:
# image: appropriate/curl:latest
# depends_on:
# - es01
# - es02
# - es03
# networks:
# - elastic
# command: "curl -v -XPUT 'es01:9200/urteile' -H 'Content-Type: application/json' -d '
# {
# 'mappings': {
# 'properties': {
# 'date': {
# 'type': 'date'
# }
# }
# }
# }
# '"

  #web:
   # build: .
   # ports:
    # - 8000:8000
    #depends_on:
    # - es01
    # - es02
    # - es03
    #networks:
    # - elastic

volumes:
  data01:
    driver: local
  data02:
    driver: local
  data03:
    driver: local

networks:
  elastic:
    driver: bridge

```

And docker info:

```auto
Server:
 Containers: 6
  Running: 3
  Paused: 0
  Stopped: 3
 Images: 185
 Server Version: 19.03.12
 Storage Driver: overlay
  Backing Filesystem: extfs
  Supports d_type: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc nvidia
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.7.2-kd-cluster
 Operating System: Debian GNU/Linux 9 (stretch)
 OSType: linux
 Architecture: x86_64
 CPUs: 32
 Total Memory: 125.8GiB
 Name: dpl01
 ID: KBGO:2E6L:NIHR:UQAL:K5CN:XWBI:R7TK:WWZF:MZBT:BCHE:HUQW:UKKM
 Docker Root Dir: /data/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

```

---

<div class="post-metadata">

**Author:** ![Stimmot](https://avatars.discourse-cdn.com/v4/letter/s/46a35a/32.png) [@Stimmot](https://discuss.elastic.co/u/Stimmot)\
**Post date:** [September 14, 2020, 8:56am UTC](https://discuss.elastic.co/t/elasticsearch-docker-flood-stage-disk-watermark-95-exceeded/248479/2 "2020-09-14T08:56:03Z")

</div>

I found the solution. The problem has to do with the disk usage in total as described in the answer from sastorsl here:

> <https://stackoverflow.com/questions/33369955/low-disk-watermark-exceeded-on>

I was working on a cluster the storage of which was 98% used, still there were 400GB free, but Elasticsearch only looks at the percentages, thus shutting down any write permissions of indices.

The solution is to manually set the watermarks after the nodes have started (setting them in the elasticsearch.yml didn't work for some reason):

```auto
curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_cluster/settings -d '{ "transient": { "cluster.routing.allocation.disk.threshold_enabled": false } }'
curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'

```

Of course you have to put in your index names.  
After that, they will be writable again.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 14, 2020, 9:13am UTC](https://discuss.elastic.co/t/elasticsearch-docker-flood-stage-disk-watermark-95-exceeded/248479/3 "2020-09-14T09:13:11Z")

</div>

That is a bit dangerous as you may run out of space, which in turn could corrupt your indices. It would probably be better to revise the levels rather than disable them.

---

<div class="post-metadata">

**Author:** ![Stimmot](https://avatars.discourse-cdn.com/v4/letter/s/46a35a/32.png) [@Stimmot](https://discuss.elastic.co/u/Stimmot)\
**Post date:** [September 14, 2020, 9:53am UTC](https://discuss.elastic.co/t/elasticsearch-docker-flood-stage-disk-watermark-95-exceeded/248479/4 "2020-09-14T09:53:30Z")

</div>

Okay, thank you for the note. Could you tell me how to do this with curl commands?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 12, 2020, 9:53am UTC](https://discuss.elastic.co/t/elasticsearch-docker-flood-stage-disk-watermark-95-exceeded/248479/5 "2020-10-12T09:53:33Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.