# Is it possible to balance shard with size in consideration?

**URL:** <https://discuss.elastic.co/t/is-it-possible-to-balance-shard-with-size-in-consideration/214296>\
**Category:** Elasticsearch\
**Created:** [January 8, 2020, 7:53pm UTC](https://discuss.elastic.co/t/is-it-possible-to-balance-shard-with-size-in-consideration/214296 "2020-01-08T19:53:10Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![linkerc](https://avatars.discourse-cdn.com/v4/letter/l/13edae/32.png) [@linkerc](https://discuss.elastic.co/u/linkerc)\
**Post date:** [January 8, 2020, 7:53pm UTC](https://discuss.elastic.co/t/is-it-possible-to-balance-shard-with-size-in-consideration/214296/1 "2020-01-08T19:53:10Z")

</div>

Is this possible?  
I know the default is simply shard count, which is important.  
But not all shards have the same size.  
Is it possible to configure size as a secondary consideration?  
Meaning pick a larger (or smaller) size shard when moving between nodes based on the capacity of the 2 nodes.  
There might be slightly more overhead but It will help balancing storage size across cluster much faster when adding new nodes.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [January 8, 2020, 8:24pm UTC](https://discuss.elastic.co/t/is-it-possible-to-balance-shard-with-size-in-consideration/214296/2 "2020-01-08T20:24:40Z")

</div>

Elasticsearch does take account of the capacity of each node when relocating shards, ensuring that it doesn't start a relocation that would breach a disk watermark.

The most time-consuming bit of rebalancing a cluster is moving the data onto the new node. This means it doesn't really matter if it moves a single large shard or two shards each half the size, since the amount of data (and therefore the time it takes) is the same.

---

<div class="post-metadata">

**Author:** ![linkerc](https://avatars.discourse-cdn.com/v4/letter/l/13edae/32.png) [@linkerc](https://discuss.elastic.co/u/linkerc)\
**Post date:** [January 8, 2020, 9:41pm UTC](https://discuss.elastic.co/t/is-it-possible-to-balance-shard-with-size-in-consideration/214296/3 "2020-01-08T21:41:26Z")

</div>

Thanks for the reply.  
I don't mean the capacity of the node itself.  
I mean moving larger shard instead of smaller shard if the current node is using more capacity.  
It's more of a relative decision. Breaching watermark is not a concern for me.  
I add nodes when capacity is \> 80%.

Looking at overall node capacity is fine. I am looking for a way to naturally balance shards across my cluster.  
I have many indices that don't have the same capacity. If shard relocation can have this secondary order, it will help a lot.

Once the target node has been selected, there are many candidate shards the algorithm can pick. If say shard A is 1mb and shard B is 1GB, I'm looking for a way to "encourage" shard selection to pick B if target node has more space, etc.

The reason I'm asking for this is that disk usage for my newest node is 33%, where as all other nodes are around 45% (2gb disk each node). That node has been added more than 1 month ago.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [January 9, 2020, 7:41am UTC](https://discuss.elastic.co/t/is-it-possible-to-balance-shard-with-size-in-consideration/214296/4 "2020-01-09T07:41:34Z")

</div>

> [@linkerc](#):
>
> The reason I'm asking for this is that disk usage for my newest node is 33%, where as all other nodes are around 45% (2gb disk each node). That node has been added more than 1 month ago.

Perhaps I'm missing something. Can you explain why this is a problem?

---

<div class="post-metadata">

**Author:** ![linkerc](https://avatars.discourse-cdn.com/v4/letter/l/13edae/32.png) [@linkerc](https://discuss.elastic.co/u/linkerc)\
**Post date:** [January 9, 2020, 7:03pm UTC](https://discuss.elastic.co/t/is-it-possible-to-balance-shard-with-size-in-consideration/214296/5 "2020-01-09T19:03:28Z")

</div>

The intend of this post is not asking for a solution to an issue.  
I am asking for a potential setting/configuration that I might have overlooked.

A new node after 1 month still not balanced in term of storage suggests that size is not in the relocation consideration, hence this question.  
Is there a knob to take shard size as a secondary criterion?

Anomalies usually suggest something is off, which is why I am handling it early to prevent potential issues in the future.

I like that ES moves shards around to balance across cluster. As far as I know, it is purely shard count by default. It is hands free most of the time. But when the capacity gets high (\> 80%), that much delta becomes an issue. It could mean that I need to add nodes few weeks earlier. Little money wasted like this do add up eventually over time.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [January 9, 2020, 7:36pm UTC](https://discuss.elastic.co/t/is-it-possible-to-balance-shard-with-size-in-consideration/214296/6 "2020-01-09T19:36:53Z")

</div>

I see. No, the situation you describe is well within expectations. Elasticsearch aims to keep all nodes below the high watermark (defaults to 90%) and mostly achieves this without needing to move any shards thanks to the low watermark (defaults to 85%). If nodes are below the low watermark then their disk usage is basically ignored. It sounds like you have ample capacity: based on your disk usage numbers you might even have too many nodes right now.

---

<div class="post-metadata">

**Author:** ![linkerc](https://avatars.discourse-cdn.com/v4/letter/l/13edae/32.png) [@linkerc](https://discuss.elastic.co/u/linkerc)\
**Post date:** [January 9, 2020, 9:04pm UTC](https://discuss.elastic.co/t/is-it-possible-to-balance-shard-with-size-in-consideration/214296/7 "2020-01-09T21:04:01Z")

</div>

Thanks.  
Most of our indices are monthly; therefore, we backup and delete older month indices.  
The capacity fills up quickly. I just performed the backup delete step, hence low capacity today.

Is it recommended to fill up storage \> 80%? I assume it's a function of OS file system performance. AFAIK, free capacity \< 20% usually cause OS degradation.  
I assume the watermark is for fail safe protection.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [January 9, 2020, 10:07pm UTC](https://discuss.elastic.co/t/is-it-possible-to-balance-shard-with-size-in-consideration/214296/8 "2020-01-09T22:07:46Z")

</div>

> [@linkerc](#):
>
> AFAIK, free capacity \< 20% usually cause OS degradation.

I also think that an overfilled filesystem would perform worse, but you'll have to do your own experiments to find out exactly where the sweet spot is for your system. 90% is the default but if you'd rather Elasticsearch automatically kept its disks below 80% then by all means reduce the high watermark to 80%.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 6, 2020, 10:07pm UTC](https://discuss.elastic.co/t/is-it-possible-to-balance-shard-with-size-in-consideration/214296/9 "2020-02-06T22:07:46Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.