Understanding Disk-based Shard Allocation better

[kghbln (Karsten Hoffmeyer)]

Even after reading the docu I have no idea what to set to my "elasticsearch.yml" file to prevent the following pain which I am getting all at the same time:

low disk watermark [20gb] exceeded on ...
high disk watermark [10gb] exceeded on ...
flood stage disk watermark [5gb] exceeded on ...

My hard disk has 25 GB of free space available and is at 93%.

This setting did not work:

cluster.routing.allocation.disk.watermark.flood_stage: 99%
cluster.routing.allocation.disk.watermark.low: 95%
cluster.routing.allocation.disk.watermark.high: 97%

This setting did not work either (see above):

cluster.routing.allocation.disk.watermark.flood_stage: 5gb
cluster.routing.allocation.disk.watermark.low: 20gb
cluster.routing.allocation.disk.watermark.high: 10gb

I consider the amount of data I would like to move to Elasticsearch as marginal.

Any insights will be great. I guess I just completely misunderstood that the docu is trying to tell me.

[DavidTurner]

@kghbln

My hard disk has 25 GB of free space available and is at 93%.

If so, the settings you quote look correct. Can you share the log messages you're seeing indicating that the disk-based shard allocator is causing a problem?

[kghbln (Karsten Hoffmeyer)]

@DavidTurner

Can you share the log messages you're seeing indicating that the disk-based shard allocator is causing a problem?

Sure, see the error log

[DavidTurner]

[2019-02-10T05:32:43,078][WARN ][o.e.c.r.a.DiskThresholdMonitor] [Ql8wRCh] flood stage disk watermark [5gb] exceeded on [Ql8wRChKRb-YoZ0gz0nBDA][Ql8wRCh][/var/lib/elasticsearch/nodes/0] free: 2.4gb[0.8%], all indices on this node will be marked read-only

This indicates that you have substantially less free space than you think. Can you run df and see if it agrees with these numbers?

[kghbln (Karsten Hoffmeyer)]

@DavidTurner

This indicates that you have substantially less free space than you think. Can you run df and see if it agrees with these numbers?

This is exactly what is puzzling me: Just did a df -h and there appears to be enough space to cater for the setting.

Dateisystem    Größe Benutzt Verf. Verw% Eingehängt auf
udev            992M       0  992M    0% /dev
tmpfs           201M     22M  179M   11% /run
/dev/vda1       292G    253G   25G   92% /
tmpfs          1003M       0 1003M    0% /dev/shm
tmpfs           5,0M       0  5,0M    0% /run/lock
tmpfs          1003M       0 1003M    0% /sys/fs/cgroup
tmpfs           201M       0  201M    0% /run/user/0

Note: I am aware that I do need more disk space.

[DavidTurner]

Is Elasticsearch still running and reporting a lack of space every 30 seconds? The timestamp on the last log message you shared looks several hours old (or maybe is in a different timezone).

[kghbln (Karsten Hoffmeyer)]

Actually no. I am getting this every morning for a couple of days now. When I mitigate by unblocking with curl:

curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'

This however makes me think that it is a temporary space shortage at night, when I do backups. Hmm ... let me change something and get back here tomorrow.

[DavidTurner]

@kghbln

This however makes me think that it is a temporary space shortage at night, when I do backups

That sounds about right. Something (possibly Elasticsearch, possibly another process) is consuming nearly all your disk space and then releasing it again. Elasticsearch is rather averse to running out of disk space, so when you exceed the flood_stage watermark it takes the fairly drastic step of marking everything as read-only to protect itself.

In the long run you will need more disk space. You may be able to work around this in the short term by isolating Elasticsearch on its own filesystem (assuming it's not Elasticsearch that's consuming all the disk). If desperate you can set the flood_stage watermark to something tiny (e.g. 100mb).

[kghbln (Karsten Hoffmeyer)]

Thanks a lot for your insight.

@DavidTurner

In the long run you will need more disk space.

Indeed and I am already planning for a new server.

@DavidTurner

You may be able to work around this in the short term by isolating Elasticsearch on its own filesystem (assuming it's not Elasticsearch that's consuming all the disk). If desperate you can set the flood_stage watermark to something tiny (e.g. 100mb ).

I will try this after I got the results from what I just changed concerning the backups. Thanks for the tip!

[kghbln (Karsten Hoffmeyer)]

@kghbln

I will try this after I got the results from what I just changed concerning the backups. Thanks for the tip!

Things cannot be solved on my server since even after tweaking there is not enough disk space left to avoid high and low disk watermark warnings/issues. I need to get a bigger volume with sufficient space.

However setting flood_stage to a much lower value for the time being helps preventing the rerouting of shards from being stopped entirely due to locking down the disk for Elasticsearch:

cluster.routing.allocation.disk.watermark.flood_stage: 1gb

Other lessons learned:

Elasticsearch actually needs quite a lot for disk space to do replication so the requirements in GB suggested for high and low disk watermarks do need to be met. If you have disk space galore one can use settings with percentages and if you are tight in disk space you need to set hard values in gb.

@DavidTurner Thanks a lot for guiding me through my troubled waters. This is very much appreciated!

[DavidTurner]

You're welcome @kghbln, glad we could help.