How to handle watermark.low

Elastic Stack Elasticsearch
#1

Hi,

We recently had the case that multiple nodes of our ElasticSearch cluster running version 7.3.2 reached the low watermark threshold.
We immediately added some nodes to counteract the increase in disk usage and made a call to /_cluster/reroute.
However, shards of the low watermark nodes were not relocated until we excluded one of the nodes via _cluster setting. After that, all nodes recovered subsequently.
What would be the best practice for handling low watermark issues?
Is there a better way to start the relocation process for shards?

Best regards

#2

The 7.10 manual contains guidance on this subject which mostly applies to 7.3.2 as well. In particular:

NOTE: It is normal for nodes to temporarily exceed the high watermark from time to time.

It is also certainly normal for nodes to exceed the low watermark for extended periods, and you don't need to take any action if you see it happening.

The only bit of that guidance that does not apply to 7.3.2 is this sentence:

When disk usage on the affected node drops below the high watermark, Elasticsearch automatically removes the write block.

#3

Hello David,

thanks for your answer. We were wondering why the cluster changed its state to yellow when the low watermarks were reached.
Also, the unassigned shards were not relocated/created to newly launched nodes until we explicitly excluded one of the old nodes.
Is this a normal behaviour?

#4

It is certainly not normal for the cluster health to go yellow just because it hit the low watermark, so I think there are other things at work that you haven't described. The cluster allocation explain API is the recommended way to determine why there are unassigned shards.

Yes, that is normal behaviour and is described in the docs to which I linked earlier. Elasticsearch doesn't move shards around simply because some nodes are over the low watermark.

#5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.