# Watermark and shard allocation

**URL:** <https://discuss.elastic.co/t/watermark-and-shard-allocation/89785>\
**Category:** Elasticsearch\
**Created:** [June 16, 2017, 11:05pm UTC](https://discuss.elastic.co/t/watermark-and-shard-allocation/89785 "2017-06-16T23:05:46Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![nevermind](https://avatars.discourse-cdn.com/v4/letter/n/53a042/32.png) [@nevermind](https://discuss.elastic.co/u/nevermind)\
**Post date:** [June 16, 2017, 11:05pm UTC](https://discuss.elastic.co/t/watermark-and-shard-allocation/89785/1 "2017-06-16T23:05:46Z")

</div>

We have a fleet of approximately 200 nodes. We use ES 1.7.4 version.  
Watermark.low equals to 80% and high to 85%.  
Around 5-8% of the fleet has Disk Usage between 80% and 85% and 5% of nodes - around 20-30% of DU. When a node with ~79-83% Disk Usage goes down and then up, shards are not getting allocated back to the node and are getting spread across the fleet. Current delayed allocation timeout is 10 minutes.

So it looks like when a node comes back, ES tries to add a biggest shard from that node back to it, but it sums up the current disk usage with that shard size, which exceeds low watermark threshold and doesn't assign anything back to the node.

Is it the expected behavior? I thought ES shouldn't sum up the Disk Usage with a shard size which is already part of a Disk Usage, should it?

Thanks!

---

<div class="post-metadata">

**Author:** ![vb3](https://avatars.discourse-cdn.com/v4/letter/v/7993a0/32.png) [@vb3](https://discuss.elastic.co/u/vb3)\
**Post date:** [June 19, 2017, 7:52pm UTC](https://discuss.elastic.co/t/watermark-and-shard-allocation/89785/2 "2017-06-19T19:52:10Z")

</div>

I wonder if it is somewhat related to [Replica shards stuck in Initialization phase](https://discuss.elastic.co/t/replica-shards-stuck-in-initialization-phase/48448) ?

@warkolm can you comment on this? Seems like a bug in 1.7.4

Thank you.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 17, 2017, 7:52pm UTC](https://discuss.elastic.co/t/watermark-and-shard-allocation/89785/3 "2017-07-17T19:52:39Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.