I have a problem with my Elasticsearch nodes running in a docker environment. I'm starting them up with docker-compose and after a few minutes they tell me:
flood stage disk watermark [95%] exceeded
I'm running it on a cluster with rather high storage capacity and I already tried to increase the watermark settings in the elasticsearch.yml file, but I still get the error. Maybe it has to do with the size of the docker containers.
Does anyone know what could be the problem? Any help is much appreciated.
The docker-compose.yml for reference:
version: '3.4'
services:
es01:
image: docker.elastic.co/elasticsearch/elasticsearch:7.8.1
container_name: es01
environment:
#- discovery.type=single-node
- node.name=es01
- cluster.name=es-docker-cluster
- discovery.seed_hosts=es02,es03
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=true
- xpack.security.enabled=false
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- data01:/usr/share/elasticsearch/data
ports:
- 9200:9200
networks:
- elastic
es02:
image: docker.elastic.co/elasticsearch/elasticsearch:7.8.1
container_name: es02
environment:
- node.name=es02
- cluster.name=es-docker-cluster
- discovery.seed_hosts=es01,es03
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- data02:/usr/share/elasticsearch/data
networks:
- elastic
es03:
image: docker.elastic.co/elasticsearch/elasticsearch:7.8.1
container_name: es03
environment:
- node.name=es03
- cluster.name=es-docker-cluster
- discovery.seed_hosts=es01,es02
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- data03:/usr/share/elasticsearch/data
networks:
- elastic
kib01:
image: docker.elastic.co/kibana/kibana:7.8.1
container_name: kib01
depends_on:
- es01
- es02
- es03
ports:
- 5601:5601
environment:
ELASTICSEARCH_URL: http://es01:9200
ELASTICSEARCH_HOSTS: http://es01:9200
networks:
- elastic
client:
image: appropriate/curl:latest
depends_on:
- es01
- es02
- es03
networks:
- elastic
command: sh -c "curl es01:9200 && curl kib01:5601"
dash_app:
build: .
ports:
- 0.0.0.0:8050:8050
depends_on:
- es01
- es02
- es03
- kib01
networks:
- elastic
#mapping:
# image: appropriate/curl:latest
# depends_on:
# - es01
# - es02
# - es03
# networks:
# - elastic
# command: "curl -v -XPUT 'es01:9200/urteile' -H 'Content-Type: application/json' -d '
# {
# 'mappings': {
# 'properties': {
# 'date': {
# 'type': 'date'
# }
# }
# }
# }
# '"
#web:
# build: .
# ports:
# - 8000:8000
#depends_on:
# - es01
# - es02
# - es03
#networks:
# - elastic
volumes:
data01:
driver: local
data02:
driver: local
data03:
driver: local
networks:
elastic:
driver: bridge
And docker info:
Server:
Containers: 6
Running: 3
Paused: 0
Stopped: 3
Images: 185
Server Version: 19.03.12
Storage Driver: overlay
Backing Filesystem: extfs
Supports d_type: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc nvidia
Default Runtime: runc
Init Binary: docker-init
containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 5.7.2-kd-cluster
Operating System: Debian GNU/Linux 9 (stretch)
OSType: linux
Architecture: x86_64
CPUs: 32
Total Memory: 125.8GiB
Name: dpl01
ID: KBGO:2E6L:NIHR:UQAL:K5CN:XWBI:R7TK:WWZF:MZBT:BCHE:HUQW:UKKM
Docker Root Dir: /data/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false