# Questions about watermarks and cluster.routing.allocation.cluster\_concurrent\_rebalance

**URL:** https://discuss.elastic.co/t/questions-about-watermarks-and-cluster-routing-allocation-cluster-concurrent-rebalance/78896
**Category:** Elasticsearch
**Created:** [March 16, 2017, 3:56pm UTC](https://discuss.elastic.co/t/questions-about-watermarks-and-cluster-routing-allocation-cluster-concurrent-rebalance/78896 "2017-03-16T15:56:46Z")
**Posts on this page:** 5
**Page:** 1

<div class="post-metadata">

### Author: ![lm\_es](https://avatars.discourse-cdn.com/v4/letter/l/e99b99/32.png) [@lm\_es](https://discuss.elastic.co/u/lm_es)
#### Post date: [March 16, 2017, 3:56pm UTC](https://discuss.elastic.co/t/questions-about-watermarks-and-cluster-routing-allocation-cluster-concurrent-rebalance/78896/1 "2017-03-16T15:56:46Z")

</div>

I have quick questions about two properties:

For **cluster.routing.allocation.cluster\_concurrent\_rebalance** , I see in the [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/shards-allocation.html#_shard_rebalancing_settings) I see that the default is 2. Looking at source code I see that for unlimited is "-1". My question is: what does increasing the setting by 1 mean in terms of rebalance performance? To get "significantly faster" than 2, do you have to move it to something like 4 or 10 (or 20?).

For **low/high watermarks** , in the [documentation] ([https://www.elastic.co/guide/en/elasticsearch/reference/current/disk-allocator.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/disk-allocator.html)) I see the default for low is 85% and high is 90%. If we want to set our low watermark to 95%, do you have any recommendations on what to set a high watermark? Setting it to either the same as the low watermark (95%) or the same range as defaults (+5% so 100%) both seem problematic.

Thank you.

---

<div class="post-metadata">

### Author: ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)
#### Post date: [March 16, 2017, 7:08pm UTC](https://discuss.elastic.co/t/questions-about-watermarks-and-cluster-routing-allocation-cluster-concurrent-rebalance/78896/2 "2017-03-16T19:08:26Z")

</div>

> [@lm\_es](#):
>
> My question is: what does increasing the setting by 1 mean in terms of rebalance performance?

You really have to try it on your cluster and see. IIRC the default is pretty conservative. If you are relocating you can try raising it and see.

> [@lm\_es](#):
>
> If we want to set our low watermark to 95%, do you have any recommendations on what to set a high watermark?

I wouldn't raise it unless I had very, very big disks. There are operations in Elasticsearch that temporarily increase the size on disk of the index quite a bit (`_force_merge` being the biggest offender) so I'd make sure I had plenty of head room.

---

<div class="post-metadata">

### Author: ![lm\_es](https://avatars.discourse-cdn.com/v4/letter/l/e99b99/32.png) [@lm\_es](https://discuss.elastic.co/u/lm_es)
#### Post date: [March 16, 2017, 9:30pm UTC](https://discuss.elastic.co/t/questions-about-watermarks-and-cluster-routing-allocation-cluster-concurrent-rebalance/78896/3 "2017-03-16T21:30:58Z")

</div>

Thanks for your helpful response. I have a quick follow-up:

> [@nik9000](#):
>
> I wouldn't raise it unless I had very, very big disks.

Our machines each have 3 x 1 TB EBS volumes. Based on your experience does that qualify as "very big" and does it change your recommendation?

Thanks again!

---

<div class="post-metadata">

### Author: ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)
#### Post date: [March 16, 2017, 10:10pm UTC](https://discuss.elastic.co/t/questions-about-watermarks-and-cluster-routing-allocation-cluster-concurrent-rebalance/78896/4 "2017-03-16T22:10:14Z")

</div>

I suspect it does qualify. I'd make sure I had 3x the space of you largest shard free.Something in that ballpark. More would be safer but 3x ought to be enough if you aren't doing a lot of `_force_merge` calls. Running out of free space is pretty horrible.

Stuff like long scroll operations on indexes that are receiving updates will cause extra disk usage too, but it probably won't make a big difference unless the scroll goes very long indeed.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 13, 2017, 10:10pm UTC](https://discuss.elastic.co/t/questions-about-watermarks-and-cluster-routing-allocation-cluster-concurrent-rebalance/78896/5 "2017-04-13T22:10:34Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
