# ES Constantly reballancing after restart **URL:** \ **Category:** Elasticsearch\ **Created:** [October 26, 2016, 3:01am UTC](https://discuss.elastic.co/t/es-constantly-reballancing-after-restart/63956 "2016-10-26T03:01:25Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![eperry](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eperry/32/551_2.png) [@eperry](https://discuss.elastic.co/u/eperry)\ **Post date:** [October 26, 2016, 3:01am UTC](https://discuss.elastic.co/t/es-constantly-reballancing-after-restart/63956/1 "2016-10-26T03:01:25Z")
So, I did a rolling restart freezing the cluster then restarting, but afterwards it has been rebalancing for over 24 hours. where it was stable prior to the restart So some details I have 9 data nodes, with about 3000 shards ( 6 indexes aday ) with over 800GB and receiving 10K new messages aver second. (3 master and 2 client nodes as well) HEAP SIze 30GB, and each server has 16 to 24 CPU's 90GB Ram, With a 10GB Network and a 4TB lun spead over +40 Disks. So I am wondering, if I should tweak the rebalancing thresh-hold , I have already tweaked the speed of the recovery > { > "transient" : { > "indices.recovery.max\_bytes\_per\_sec": "8000mb", > "indices.recovery.concurrent\_streams": 5, > "cluster.routing.allocation.node\_concurrent\_recoveries": 6, > "cluster.routing.allocation.cluster\_concurrent\_rebalance": 20, > "index.unassigned.node\_left.delayed\_timeout": "30m", > "cluster.routing.allocation.allow\_rebalance" : "indices\_all\_active" > } > } My freeze looks like: > { > "transient": { > "cluster.routing.allocation.enable": "none" > } > } My load average is about 3, and IO wait is constant around 5% which is that during a stable time, and there is no more then a 1 or 2 IO Queue so does not look like any issues. Any thoughts? ---
**Author:** ![chenjinyuan87](https://avatars.discourse-cdn.com/v4/letter/c/bb73d2/32.png) [@chenjinyuan87](https://discuss.elastic.co/u/chenjinyuan87)\ **Post date:** [October 26, 2016, 3:31am UTC](https://discuss.elastic.co/t/es-constantly-reballancing-after-restart/63956/2 "2016-10-26T03:31:28Z")
In our cluster, the cluster.routing.allocation is always disabled. When restarting a node, i manually allocate the shards to the original node. The reason is i found after enabling reallocation, some shards will be allocated to other nodes rather than the original(restarting) one. Is this the same situation for you? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 26, 2016, 9:08am UTC](https://discuss.elastic.co/t/es-constantly-reballancing-after-restart/63956/3 "2016-10-26T09:08:07Z")
> [@chenjinyuan87](#): > > When restarting a node, i manually allocate the shards to the original node. > > The reason is i found after enabling reallocation, some shards will be allocated to other nodes rather than the original(restarting) one. Why would you do that? Who cares where the shards are? ---
**Author:** ![chenjinyuan87](https://avatars.discourse-cdn.com/v4/letter/c/bb73d2/32.png) [@chenjinyuan87](https://discuss.elastic.co/u/chenjinyuan87)\ **Post date:** [October 26, 2016, 11:32am UTC](https://discuss.elastic.co/t/es-constantly-reballancing-after-restart/63956/4 "2016-10-26T11:32:38Z")
If the shard is allocate to a different node, the whole shard needs to be recovered. ie. if the shard size is 1T, than 1T data is recovered from the primary shard. It's too much slow for me. Especially when rolling restart. And if i allocate it to the original one, the recovery is fast. Only a small part of the shard needs to be recovery. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 26, 2016, 12:23pm UTC](https://discuss.elastic.co/t/es-constantly-reballancing-after-restart/63956/5 "2016-10-26T12:23:00Z")
> [@chenjinyuan87](#): > > ie. if the shard size is 1T, than 1T data is recovered from the primary shard. Then reduce your shard size. 1TB is _way_ too big. What version are you on? ---
**Author:** ![eperry](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eperry/32/551_2.png) [@eperry](https://discuss.elastic.co/u/eperry)\ **Post date:** [October 26, 2016, 12:39pm UTC](https://discuss.elastic.co/t/es-constantly-reballancing-after-restart/63956/6 "2016-10-26T12:39:43Z")
No my shards are like 10 to 20GB , that is an interesting idea to manually rebalance them but I think that is too much work. IMH I tried a cluster\_concurrent\_rebalance the system becomes pretty bogged down. @warkolm Warkolm, maybe I should think of it in a different way, it is not the number of shards recover which is my problem but the size of them? I mean my indexes are at 24, and not really seeing any CPU problems. Should I maybe go 50 or 100. Of course this is a production cluster so I would have to set up some kind of test but what do you think? I agree, shards at 1TB is way to large. your search times must be painfully slow @chenjinyuan87 ---
**Author:** ![chenjinyuan87](https://avatars.discourse-cdn.com/v4/letter/c/bb73d2/32.png) [@chenjinyuan87](https://discuss.elastic.co/u/chenjinyuan87)\ **Post date:** [October 27, 2016, 2:18am UTC](https://discuss.elastic.co/t/es-constantly-reballancing-after-restart/63956/7 "2016-10-27T02:18:22Z")
> [@warkolm](#): > > Then reduce your shard size. 1TB is _way_ too big. > What version are you on? We have 4TB data, the \_all field is disabled already. I don't know what else can i do? Currently it's 1.3.4. And I'm planning to upgrade it to 5.0. But i need to upgrade my custom plugins first. ---
**Author:** ![chenjinyuan87](https://avatars.discourse-cdn.com/v4/letter/c/bb73d2/32.png) [@chenjinyuan87](https://discuss.elastic.co/u/chenjinyuan87)\ **Post date:** [October 27, 2016, 2:44am UTC](https://discuss.elastic.co/t/es-constantly-reballancing-after-restart/63956/8 "2016-10-27T02:44:26Z")
@eperry @warkolm In my opinion, too much shards also slow down the search. So i think 5 shards is a good choice.... I think es should make a better reallocation strategy, make sure the shards are allocated to the original node? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:09pm UTC](https://discuss.elastic.co/t/es-constantly-reballancing-after-restart/63956/9 "2017-07-05T22:09:15Z")