# 6.4.2 All nodes except one crosses watermark level

**URL:** <https://discuss.elastic.co/t/6-4-2-all-nodes-except-one-crosses-watermark-level/157652>\
**Category:** Elasticsearch\
**Created:** [November 21, 2018, 8:35am UTC](https://discuss.elastic.co/t/6-4-2-all-nodes-except-one-crosses-watermark-level/157652 "2018-11-21T08:35:47Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Neha\_Sharma1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/neha_sharma1/32/36504_2.png) [@Neha\_Sharma1](https://discuss.elastic.co/u/Neha_Sharma1)\
**Post date:** [November 21, 2018, 8:35am UTC](https://discuss.elastic.co/t/6-4-2-all-nodes-except-one-crosses-watermark-level/157652/1 "2018-11-21T08:35:48Z")

</div>

I have 4 Data nodes in my cluster with size 2.5TB. Approx 250Gb data is added to the cluster every day. The lower disk watermark is 85%(default). We are facing an issue where replicas are not being assigned to any node.  
The watermark level of each node is as follows:  
Node1 : 86%  
Node2 : 85%  
Node3 : 89%  
Node4 : 90%

All nodes have crossed their watermark level except Node2.

Why wasn't the load distributed across all nodes and all should have crossed their watermark level? Why only Node2 stopped accepting shards/data while others continued indexing data on their nodes?

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [November 21, 2018, 9:20am UTC](https://discuss.elastic.co/t/6-4-2-all-nodes-except-one-crosses-watermark-level/157652/2 "2018-11-21T09:20:40Z")

</div>

> [@Neha\_Sharma1](#):
>
> Why wasn't the load distributed across all nodes and all should have crossed their watermark level?

I am confused by this question. All of the nodes _have_ crossed the 85% watermark.

> [@Neha\_Sharma1](#):
>
> Why only Node2 stopped accepting shards/data while others continued indexing data on their nodes?

There are three disk-based watermark levels: `low`, `high` and `flood_stage`. Once a node crosses the `low` watermark it stops accepting new shards, but continues to write to any shards that it currently has. Once it crosses the `high` watermark it starts trying to relocate its shards elsewhere, but continues to write to any shards that it currently has while the relocation is ongoing. Once it crosses the `flood_stage` watermark it marks all its indices as read only and starts rejecting indexing traffic too. The documentation has more details: [Disk-based shard allocation | Elasticsearch Guide [8.11] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/disk-allocator.html)

---

<div class="post-metadata">

**Author:** ![Neha\_Sharma1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/neha_sharma1/32/36504_2.png) [@Neha\_Sharma1](https://discuss.elastic.co/u/Neha_Sharma1)\
**Post date:** [November 21, 2018, 9:53am UTC](https://discuss.elastic.co/t/6-4-2-all-nodes-except-one-crosses-watermark-level/157652/3 "2018-11-21T09:53:10Z")

</div>

@DavidTurner Thanks 🙂

> [@DavidTurner](#):
>
> I am confused by this question. All of the nodes _have_ crossed the 85% watermark.

What I meant was that Node2 never crossed 85% while others did.

But your answer explains the watermarking clearly and I was able to find the problem in my system.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 19, 2018, 9:53am UTC](https://discuss.elastic.co/t/6-4-2-all-nodes-except-one-crosses-watermark-level/157652/4 "2018-12-19T09:53:12Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.