# Share Rebalancing on large clusters (2.4)
**URL:** https://discuss.elastic.co/t/share-rebalancing-on-large-clusters-2-4/68957
**Category:** Elasticsearch
**Created:** [December 14, 2016, 2:18am UTC](https://discuss.elastic.co/t/share-rebalancing-on-large-clusters-2-4/68957 "2016-12-14T02:18:06Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@eperry](https://discuss.elastic.co/u/eperry)
#### Post date: [December 14, 2016, 2:18am UTC](https://discuss.elastic.co/t/share-rebalancing-on-large-clusters-2-4/68957/1 "2016-12-14T02:18:06Z")
Hey guys got a question.
I have a cluster with over 20k total shards (each index is 30 shards + 1 replication and is oh about 300GB each) on 18 Data nodes with ~24 cores on each node. oh and we are indexing 10K message per second all day long (about 1TB a day of data)
1. When \_open a couple of indexes at a time the cluster re balances for a while
2. when doing maintenance on a node it takes for ever for it to rebalance
When the re-routing/rebalancing/recovering is happening my indexing slows way down.
So here are my questions
1. I know there are Heuristics on when the cluster chooses to re balance but I don't understand the meaning of the numbers so I am afraid to touch them. Any resources that can help describe this better (or should I look somewhere else)
[https://www.elastic.co/guide/en/elasticsearch/reference/master/shards-allocation.html#\_shard\_balancing\_heuristics](https://www.elastic.co/guide/en/elasticsearch/reference/master/shards-allocation.html#_shard_balancing_heuristics)
1. I have looked at the Thread Queues but don't see any threads being maxed out during the re-balancing. and I have played with the concurrent load balancing settings at the cluster level. but doing it slow (concurrent rebalancing 2 or three) or fast at +30 seems to have the same impact.
[https://www.elastic.co/guide/en/elasticsearch/reference/master/shards-allocation.html#\_shard\_allocation\_settings](https://www.elastic.co/guide/en/elasticsearch/reference/master/shards-allocation.html#_shard_allocation_settings)
---
### Author:  [@eperry](https://discuss.elastic.co/u/eperry)
#### Post date: [December 14, 2016, 2:20am UTC](https://discuss.elastic.co/t/share-rebalancing-on-large-clusters-2-4/68957/2 "2016-12-14T02:20:16Z")
```auto
curl -XPUT $HOSTNAME:9200/_cluster/settings -d '{
"transient" : {
"indices.recovery.max_bytes_per_sec": "5000mb",
"indices.recovery.concurrent_streams": 8,
"cluster.routing.allocation.node_concurrent_recoveries": 8,
"cluster.routing.allocation.node_initial_primaries_recoveries": 4,
"cluster.routing.allocation.cluster_concurrent_rebalance": 8,
"index.unassigned.node_left.delayed_timeout": "1m",
"index.refresh_interval" : "5s",
"cluster.routing.allocation.enable" : "all",
"cluster.routing.allocation.allow_rebalance" : "always"
}
}'
```
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [December 14, 2016, 2:24am UTC](https://discuss.elastic.co/t/share-rebalancing-on-large-clusters-2-4/68957/3 "2016-12-14T02:24:00Z")
> [@eperry](#):
>
> each index is 30 shards + 1 replication and is oh about 300GB each
Is the index is 300GB, or the shard?
---
### Author:  [@eperry](https://discuss.elastic.co/u/eperry)
#### Post date: [December 14, 2016, 2:28am UTC](https://discuss.elastic.co/t/share-rebalancing-on-large-clusters-2-4/68957/4 "2016-12-14T02:28:22Z")
The Index, ranges from 100GB - 300GB depending on the index. ( ~5 different indexes adding up to the 1TB a day)
the share should be be about 10GB for the 300GB index (300GB / 30 shards = 10gb) right?
we are thinking about adding shards because we are adding hosts.
We are ok with slow searches but can't deal with a backlog of indexing during recovery
---
### Author:  [@eperry](https://discuss.elastic.co/u/eperry)
#### Post date: [December 22, 2016, 5:32pm UTC](https://discuss.elastic.co/t/share-rebalancing-on-large-clusters-2-4/68957/5 "2016-12-22T17:32:58Z")
no idea's?
well, I did find one issue as that I was CPU/IO Bound. I added 4 more luns and went from 50Mbps to 500Mbps so that will help once I finish spliting the 15 nodes.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [January 19, 2017, 5:33pm UTC](https://discuss.elastic.co/t/share-rebalancing-on-large-clusters-2-4/68957/6 "2017-01-19T17:33:20Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.