We recently had the case that multiple nodes of our ElasticSearch cluster running version 7.3.2 reached the low watermark threshold.
We immediately added some nodes to counteract the increase in disk usage and made a call to /_cluster/reroute.
However, shards of the low watermark nodes were not relocated until we excluded one of the nodes via _cluster setting. After that, all nodes recovered subsequently.
What would be the best practice for handling low watermark issues?
Is there a better way to start the relocation process for shards?
thanks for your answer. We were wondering why the cluster changed its state to yellow when the low watermarks were reached.
Also, the unassigned shards were not relocated/created to newly launched nodes until we explicitly excluded one of the old nodes.
Is this a normal behaviour?
It is certainly not normal for the cluster health to go yellow just because it hit the low watermark, so I think there are other things at work that you haven't described. The cluster allocation explain API is the recommended way to determine why there are unassigned shards.
Yes, that is normal behaviour and is described in the docs to which I linked earlier. Elasticsearch doesn't move shards around simply because some nodes are over the low watermark.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.