Having issues with low watermark / disk space

Elastic Stack Elasticsearch
1

Hi
I have been struggling with low watermark that keeps getting hit on some nodes in my cluster.
I have a 6 node cluster, that unfortunatly have different diskspace. 3 nodes have 5 TB and 3 nodes have 10TB.

The problem is that the three nodes with less disk hits the low watermark, which in turn stops upgrades and patches from automatically install and update.
the three nodes with large disks has more than enoght disk space.

Is there any setting i can set to fix this? If I raise the disk space, it only moves more indices from other nodes down to the "low disk" nodes. i have tried changing the watermarks, but it always just crosses the threshold anyway. so if I set low watermark to 20%, it will fill up to 19.9%. , if I set to 300GB -> it fills upp to 298GB and so on...
so whatever the value, it always fills it up just enought to cross the threshold.

2 
shards disk.indices disk.used disk.avail disk.total disk.percent host 
   524        6.4tb     6.4tb      3.2tb      9.6tb         66   node_3
   370        4.3tb     4.3tb    468.1gb      4.8tb         90   node_1
   523          7tb       7tb      2.6tb      9.6tb         72   node_4
   524        6.7tb     6.8tb      2.8tb      9.6tb         70   node_5
   310        4.3tb     4.3tb    485.7gb      4.8tb         90   node_2
   341        4.3tb     4.3tb    481.9gb      4.8tb         90   node_1
3

This is normal, especially in a cluster with varying disk space. See these docs, particularly the bit that says

It is normal for nodes to temporarily exceed the high watermark from time to time.

This is not normal. There's no reason to avoid upgrading just because a node has exceeded the low watermark.

1 Like
4

ok, thanks for the fast reply. it says that when low watermark is use no new indices can be created.
The reason I ask is that I am going to upgrade this cluster to 8.17 soon (running 7.17.25 now) and the upgrade assistant say that it is critical to fix this.

Disk usage exceeds low watermark, which will prevent reindexing indices during upgrade. Get disk usage on all nodes below the value specified in cluster.routing.allocation.disk.watermark.low (nodes impacted:
5

Hmm ok I see, I think that's a bug in the upgrade assistant. If all nodes were over the low watermark then this'd be true, but as long as there are some nodes with free disk space then that should be fine I think. Would you report the bug?

6

oh that would explain it! great. I'll report it

1 Like