# Shard rebalance issue **URL:** \ **Category:** Elasticsearch\ **Created:** [July 22, 2019, 2:09pm UTC](https://discuss.elastic.co/t/shard-rebalance-issue/191656 "2019-07-22T14:09:34Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![vangogh94](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vangogh94/32/48826_2.png) [@vangogh94](https://discuss.elastic.co/u/vangogh94)\ **Post date:** [July 22, 2019, 2:09pm UTC](https://discuss.elastic.co/t/shard-rebalance-issue/191656/1 "2019-07-22T14:09:34Z")
Running ES v7.2.0. We excluded data from node 'es-dbs-022' using cluster.routing.allocation.exclude.\_name because of some unrelated reasons. After some time we returned the node back to cluster using the same API (setting value to null) and after that the cluster has been rebalancing for few days (is usually done in 30min), and the rebalance itself is behaving very weirdly. We have 60 data nodes, and one specific index has 60 shards (+60 replicas). Usually nodes have 1-3 shards from this index, but es-dbs-022 has ~30 of these shards. Other than this, the node has just a few shards that are not from this index. ![51](https://us1.discourse-cdn.com/elastic/original/3X/3/9/3952bb374120027bd180925f8434319a9226d5bc.png) This is very abnormal, usually after few hours of rebalancing the node has its disk full and blocks all writing. I tried looking into logs but nothing of importance is shown. I've put TRACE log level for logger.org.elasticsearch.cluster.routing.allocation and in here only useful thing i found is that based on BalancedShardsAllocator, the node has negative weight in half of log records. No other nodes have negative weight. One other potential clue is this 'THROTTLE' log (happens often, bfcIx0i\_Spm\_aatNSspIkg is es-dbs-022): `{"log":"{\"type\": \"server\", \"timestamp\": \"2019-07-22T13:16:12,653+0000\", \"level\": \"TRACE\", \"component\": \"o.e.c.r.a.a.BalancedShardsAllocator\", \"cluster.name\": \"es-research-cloud\", \"node.name\": \"es-dbm-001\", \"cluster.uuid\": \"Tnbn6gyVRUWU4p-m--4gIA\", \"node.id\": \"ATaVaYN6QZePAa5s5IMhsQ\", \"message\": \"Couldn't find shard to relocate from node [uDxmj5r8QEi-gvdSwu0HQw] to node [bfcIx0i_Spm_aatNSspIkg] allocation decision [THROTTLE]\" }\n","stream":"stdout","time":"2019-07-22T13:16:12.653302601Z"}` Anybody has any ideas? Thanks ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 22, 2019, 3:02pm UTC](https://discuss.elastic.co/t/shard-rebalance-issue/191656/2 "2019-07-22T15:02:07Z")
Looks like an issue related to [https://github.com/elastic/elasticsearch/issues/17213](https://github.com/elastic/elasticsearch/issues/17213). Try setting `index.routing.allocation.total_shards_per_node` on this index to prevent all the shards from becoming so concentrated. ---
**Author:** ![vangogh94](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vangogh94/32/48826_2.png) [@vangogh94](https://discuss.elastic.co/u/vangogh94)\ **Post date:** [July 23, 2019, 11:10am UTC](https://discuss.elastic.co/t/shard-rebalance-issue/191656/3 "2019-07-23T11:10:46Z")
At first I used `index.routing.allocation.exclude` for this index and few other big ones. This fixed the issue. We brought another node to the cluster and the same issue happened, this time i used your `index.routing.allocation.total_shards_per_node` proposition, it also worked. Same issue is now causing more problems, if i try to export batch data to the cluster, all shards from newly created index end up at the same node and the node dies because of writing overload: ![49](https://us1.discourse-cdn.com/elastic/original/3X/6/8/68a5854ef9e1a2a2525aa491c26521d772766260.png) I guess we can amend this by setting `total_shards_per_node` on index templates, but it is really frustrating. If this issue gets fixed, will it be applied to version 7.2.x? Another question - would increasing `cluster.routing.allocation.balance.index` be a good idea to try? Thanks! ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 23, 2019, 11:46am UTC](https://discuss.elastic.co/t/shard-rebalance-issue/191656/4 "2019-07-23T11:46:37Z")
It's almost certain that this will not be fixed in the 7.2.x series: 7.2.0 is already released, so there will be no new features in this series, and addressing this is a very substantial piece of work. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 23, 2019, 11:47am UTC](https://discuss.elastic.co/t/shard-rebalance-issue/191656/5 "2019-07-23T11:47:40Z")
> [@vangogh94](#): > > would increasing `cluster.routing.allocation.balance.index` be a good idea to try? You can certainly try it, but if you want to say to Elasticsearch "do not allocate more than N shards of this index to any one node or else that node will fall over" then `total_shards_per_node` is how to do that. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 20, 2019, 12:01pm UTC](https://discuss.elastic.co/t/shard-rebalance-issue/191656/6 "2019-08-20T12:01:43Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.