Questions about watermarks and cluster.routing.allocation.cluster_concurrent_rebalance

[lm_es]

I have quick questions about two properties:

For cluster.routing.allocation.cluster_concurrent_rebalance, I see in the documentation I see that the default is 2. Looking at source code I see that for unlimited is "-1". My question is: what does increasing the setting by 1 mean in terms of rebalance performance? To get "significantly faster" than 2, do you have to move it to something like 4 or 10 (or 20?).

For low/high watermarks, in the [documentation] (https://www.elastic.co/guide/en/elasticsearch/reference/current/disk-allocator.html) I see the default for low is 85% and high is 90%. If we want to set our low watermark to 95%, do you have any recommendations on what to set a high watermark? Setting it to either the same as the low watermark (95%) or the same range as defaults (+5% so 100%) both seem problematic.

Thank you.

[nik9000 (Nik Everett)]

@lm_es

My question is: what does increasing the setting by 1 mean in terms of rebalance performance?

You really have to try it on your cluster and see. IIRC the default is pretty conservative. If you are relocating you can try raising it and see.

@lm_es

If we want to set our low watermark to 95%, do you have any recommendations on what to set a high watermark?

I wouldn't raise it unless I had very, very big disks. There are operations in Elasticsearch that temporarily increase the size on disk of the index quite a bit (_force_merge being the biggest offender) so I'd make sure I had plenty of head room.

[lm_es]

Thanks for your helpful response. I have a quick follow-up:

@nik9000

I wouldn't raise it unless I had very, very big disks.

Our machines each have 3 x 1 TB EBS volumes. Based on your experience does that qualify as "very big" and does it change your recommendation?

Thanks again!

[nik9000 (Nik Everett)]

I suspect it does qualify. I'd make sure I had 3x the space of you largest shard free.Something in that ballpark. More would be safer but 3x ought to be enough if you aren't doing a lot of _force_merge calls. Running out of free space is pretty horrible.

Stuff like long scroll operations on indexes that are receiving updates will cause extra disk usage too, but it probably won't make a big difference unless the scroll goes very long indeed.