# How to handle watermark.low

**URL:** https://discuss.elastic.co/t/how-to-handle-watermark-low/258955
**Category:** Elasticsearch
**Created:** [December 17, 2020, 7:59am UTC](https://discuss.elastic.co/t/how-to-handle-watermark-low/258955 "2020-12-17T07:59:45Z")
**Posts on this page:** 5
**Page:** 1

<div class="post-metadata">

### Author: ![Kxn7](https://avatars.discourse-cdn.com/v4/letter/k/ba8739/32.png) [@Kxn7](https://discuss.elastic.co/u/Kxn7)
#### Post date: [December 17, 2020, 7:59am UTC](https://discuss.elastic.co/t/how-to-handle-watermark-low/258955/1 "2020-12-17T07:59:45Z")

</div>

Hi,

We recently had the case that multiple nodes of our ElasticSearch cluster running version 7.3.2 reached the low watermark threshold.  
We immediately added some nodes to counteract the increase in disk usage and made a call to /\_cluster/reroute.  
However, shards of the low watermark nodes were not relocated until we excluded one of the nodes via \_cluster setting. After that, all nodes recovered subsequently.  
What would be the best practice for handling low watermark issues?  
Is there a better way to start the relocation process for shards?

Best regards

---

<div class="post-metadata">

### Author: ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [December 17, 2020, 8:42am UTC](https://discuss.elastic.co/t/how-to-handle-watermark-low/258955/2 "2020-12-17T08:42:18Z")

</div>

The 7.10 manual contains [guidance on this subject](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cluster.html#disk-based-shard-allocation) which mostly applies to 7.3.2 as well. In particular:

> **NOTE** : It is normal for nodes to temporarily exceed the high watermark from time to time.

It is also certainly normal for nodes to exceed the low watermark for extended periods, and you don't need to take any action if you see it happening.

The only bit of that guidance that does not apply to 7.3.2 is this sentence:

> When disk usage on the affected node drops below the high watermark, Elasticsearch automatically removes the write block.

---

<div class="post-metadata">

### Author: ![Kxn7](https://avatars.discourse-cdn.com/v4/letter/k/ba8739/32.png) [@Kxn7](https://discuss.elastic.co/u/Kxn7)
#### Post date: [December 23, 2020, 3:03pm UTC](https://discuss.elastic.co/t/how-to-handle-watermark-low/258955/3 "2020-12-23T15:03:29Z")

</div>

Hello David,

thanks for your answer. We were wondering why the cluster changed its state to yellow when the low watermarks were reached.  
Also, the unassigned shards were not relocated/created to newly launched nodes until we explicitly excluded one of the old nodes.  
Is this a normal behaviour?

---

<div class="post-metadata">

### Author: ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [December 23, 2020, 4:02pm UTC](https://discuss.elastic.co/t/how-to-handle-watermark-low/258955/4 "2020-12-23T16:02:13Z")

</div>

> [@Kxn7](#):
>
> We were wondering why the cluster changed its state to yellow when the low watermarks were reached.

It is certainly not normal for the cluster health to go `yellow` just because it hit the low watermark, so I think there are other things at work that you haven't described. The [cluster allocation explain API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-allocation-explain.html) is the recommended way to determine why there are unassigned shards.

> [@Kxn7](#):
>
> Also, the unassigned shards were not relocated/created to newly launched nodes until we explicitly excluded one of the old nodes.

Yes, that is normal behaviour and is described in the docs to which I linked earlier. Elasticsearch doesn't move shards around simply because some nodes are over the low watermark.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [January 20, 2021, 4:02pm UTC](https://discuss.elastic.co/t/how-to-handle-watermark-low/258955/5 "2021-01-20T16:02:29Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
