# Re-balancing shard allocation **URL:** \ **Category:** Elasticsearch\ **Created:** [May 22, 2018, 6:26am UTC](https://discuss.elastic.co/t/re-balancing-shard-allocation/132726 "2018-05-22T06:26:37Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![hamid\_bayat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hamid_bayat/32/74056_2.png) [@hamid\_bayat](https://discuss.elastic.co/u/hamid_bayat)\ **Post date:** [May 22, 2018, 6:26am UTC](https://discuss.elastic.co/t/re-balancing-shard-allocation/132726/1 "2018-05-22T06:26:37Z")
we have cluster with 100 node and our index name are base on date. with elastic default options, the situation is like this: the shard of index that begin with 2017-June are in first 10 node the shard of index that begin with 2017-May are in second 10 node the shard of index that begin with 2017-April are in third 10 node and so on for every month. the problem is workload on the 10 node that have last month index are very high and we want to distribute the shards and work load. I think elastic have option for this situation: cluster.routing.allocation.balance.index cluster.routing.allocation.balance.shard with it get a float number. Am I correct with this? how much i should raise the float number? any explanation will be appreciate thanks. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 22, 2018, 6:29am UTC](https://discuss.elastic.co/t/re-balancing-shard-allocation/132726/2 "2018-05-22T06:29:59Z")
How did you end up with that kind of distribution in the first place? I would expect Elasticsearch to spread it out quite evenly by default. ---
**Author:** ![hamid\_bayat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hamid_bayat/32/74056_2.png) [@hamid\_bayat](https://discuss.elastic.co/u/hamid_bayat)\ **Post date:** [May 22, 2018, 6:31am UTC](https://discuss.elastic.co/t/re-balancing-shard-allocation/132726/3 "2018-05-22T06:31:57Z")
I don't know. as I said, all setting are default and we didn't change any specific options. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 22, 2018, 6:43am UTC](https://discuss.elastic.co/t/re-balancing-shard-allocation/132726/4 "2018-05-22T06:43:07Z")
How many shards do you have per node? Are you using daily indices? Can you show the configuration for one of the nodes as well as the index settings for an index on that node? ---
**Author:** ![hamid\_bayat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hamid_bayat/32/74056_2.png) [@hamid\_bayat](https://discuss.elastic.co/u/hamid_bayat)\ **Post date:** [May 22, 2018, 6:56am UTC](https://discuss.elastic.co/t/re-balancing-shard-allocation/132726/5 "2018-05-22T06:56:07Z")
yes. we use daily indices. at least 100 shards per node. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 22, 2018, 6:59am UTC](https://discuss.elastic.co/t/re-balancing-shard-allocation/132726/6 "2018-05-22T06:59:00Z")
That sounds like a very large number of potentially small shards, which can be very inefficient. Please read [this blog post](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster) for some practical guidelines around sharding and shard sizes. ---
**Author:** ![hamid\_bayat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hamid_bayat/32/74056_2.png) [@hamid\_bayat](https://discuss.elastic.co/u/hamid_bayat)\ **Post date:** [May 22, 2018, 7:04am UTC](https://discuss.elastic.co/t/re-balancing-shard-allocation/132726/7 "2018-05-22T07:04:09Z")
no. the shards at least have 50 GB data and they are not small thank you for responding but the problem is about how to distribute last month shards between 50% of nodes. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 22, 2018, 7:06am UTC](https://discuss.elastic.co/t/re-balancing-shard-allocation/132726/8 "2018-05-22T07:06:28Z")
If your shards are at least 50GB in size and you have over 1000 shards per node, that means over 50TB of data per node, is that correct? Are all these indices open? Can you perhaps provide the output of the [cluster stats API](https://www.elastic.co/guide/en/elasticsearch/reference/6.2/cluster-stats.html)? ---
**Author:** ![hamid\_bayat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hamid_bayat/32/74056_2.png) [@hamid\_bayat](https://discuss.elastic.co/u/hamid_bayat)\ **Post date:** [May 22, 2018, 7:11am UTC](https://discuss.elastic.co/t/re-balancing-shard-allocation/132726/9 "2018-05-22T07:11:19Z")
sorry. I did a mistake. 100 shards per node. yes we have 5 TB per node. ---
**Author:** ![hamid\_bayat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hamid_bayat/32/74056_2.png) [@hamid\_bayat](https://discuss.elastic.co/u/hamid_bayat)\ **Post date:** [May 22, 2018, 7:12am UTC](https://discuss.elastic.co/t/re-balancing-shard-allocation/132726/10 "2018-05-22T07:12:16Z")
I cant send you cluster stats now. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 22, 2018, 7:13am UTC](https://discuss.elastic.co/t/re-balancing-shard-allocation/132726/11 "2018-05-22T07:13:29Z")
Then it sounds like you are following best practices. 🙂 ---
**Author:** ![hamid\_bayat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hamid_bayat/32/74056_2.png) [@hamid\_bayat](https://discuss.elastic.co/u/hamid_bayat)\ **Post date:** [May 22, 2018, 7:15am UTC](https://discuss.elastic.co/t/re-balancing-shard-allocation/132726/12 "2018-05-22T07:15:19Z")
can you describe cluster.routing.allocation.balance.shard option? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 22, 2018, 7:19am UTC](https://discuss.elastic.co/t/re-balancing-shard-allocation/132726/13 "2018-05-22T07:19:11Z")
How many shards do you have per index? How large portion of indices related to June 2017 are located on this set of 10 nodes? Is there more than 1 shard for any particular index on any one node? ---
**Author:** ![hamid\_bayat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hamid_bayat/32/74056_2.png) [@hamid\_bayat](https://discuss.elastic.co/u/hamid_bayat)\ **Post date:** [May 22, 2018, 7:25am UTC](https://discuss.elastic.co/t/re-balancing-shard-allocation/132726/14 "2018-05-22T07:25:04Z")
between 5 to 10 shards according to indices. no. there is not more than 1 shard of a index on any node. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 22, 2018, 7:39am UTC](https://discuss.elastic.co/t/re-balancing-shard-allocation/132726/15 "2018-05-22T07:39:52Z")
Then those settings may not help much as they look to distribute shards of individual indices across nodes. You may need to [manually reroute](https://www.elastic.co/guide/en/elasticsearch/reference/6.2/cluster-reroute.html) indices. ---
**Author:** ![hamid\_bayat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hamid_bayat/32/74056_2.png) [@hamid\_bayat](https://discuss.elastic.co/u/hamid_bayat)\ **Post date:** [May 22, 2018, 7:43am UTC](https://discuss.elastic.co/t/re-balancing-shard-allocation/132726/16 "2018-05-22T07:43:10Z")
thanks for the answers. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [May 22, 2018, 9:27am UTC](https://discuss.elastic.co/t/re-balancing-shard-allocation/132726/17 "2018-05-22T09:27:18Z")
How many indices per month are being created? ---
**Author:** ![hamid\_bayat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hamid_bayat/32/74056_2.png) [@hamid\_bayat](https://discuss.elastic.co/u/hamid_bayat)\ **Post date:** [May 22, 2018, 11:18am UTC](https://discuss.elastic.co/t/re-balancing-shard-allocation/132726/19 "2018-05-22T11:18:03Z")
60 indices. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [May 22, 2018, 12:31pm UTC](https://discuss.elastic.co/t/re-balancing-shard-allocation/132726/20 "2018-05-22T12:31:34Z")
Wild guess: are you perhaps deleting some "old" indices just before creating the new ones each month? If so, Elasticsearch may be preferring to put the new shards where the old ones have left gaps, and if the old indices were all concentrated on 10 nodes then the new ones will be too. If it is this, then creating the new indices **without** first deleting the old ones should cause them to be allocated more evenly. ---
**Author:** ![hamid\_bayat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hamid_bayat/32/74056_2.png) [@hamid\_bayat](https://discuss.elastic.co/u/hamid_bayat)\ **Post date:** [May 23, 2018, 6:11am UTC](https://discuss.elastic.co/t/re-balancing-shard-allocation/132726/21 "2018-05-23T06:11:02Z")
> [@DavidTurner](#): > > the new indices **without** first dele thanks for responds. we don't delete any index. [Next page](https://discuss.elastic.co/t/re-balancing-shard-allocation/132726.md?page=2)