# Cluster performance when adding a node **URL:** \ **Category:** Elasticsearch\ **Created:** [January 21, 2013, 6:42pm UTC](https://discuss.elastic.co/t/cluster-performance-when-adding-a-node/10442 "2013-01-21T18:42:10Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![rore](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rore/32/399_2.png) [@rore](https://discuss.elastic.co/u/rore)\ **Post date:** [January 21, 2013, 6:42pm UTC](https://discuss.elastic.co/t/cluster-performance-when-adding-a-node/10442/1 "2013-01-21T18:42:10Z")
We encounter a lot of problems with cluster performance when trying to scale it (adding a node to the cluster and also when removing one). When you add a node the cluster starts to rebalance. The new node, once shards are allocated, starts serving requests. Since we have a cluster with constant indexing and queries, this causes an IO bottleneck on the new node. We're on EC2, not using provisioned IOps - I know that can help but wondering if there is some cluster configuration that might ease the load on cluster scaling. One thing I'm trying is reducing the cluster\_concurrent\_rebalance setting to 1 to reduce moving shards. There's also the throttling setting but I'm afraid that might impact overall performance. Any other recommendations on how to handle cluster scaling on a live cluster? The ideal situation was to be able to tell the cluster to copy shards to the new node without transferring traffic and removing old shards yet, and make the new node active only when rebalancing has finished. I guess that's not a possibility right now... -- ---
**Author:** ![simonw\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simonw_2/32/1130_2.png) [@simonw\_2](https://discuss.elastic.co/u/simonw_2)\ **Post date:** [January 21, 2013, 8:14pm UTC](https://discuss.elastic.co/t/cluster-performance-when-adding-a-node/10442/2 "2013-01-21T20:14:49Z")
Hey, I would recommend you to throttle recovery IO during adding a node to the cluster with something like "indices.recovery.max\_size\_per\_sec" : 20mb it might also make sense to set "indices.recovery.concurrent\_streams" : 1 simon On Monday, January 21, 2013 7:42:10 PM UTC+1, Rotem wrote: > We encounter a lot of problems with cluster performance when trying to > scale it (adding a node to the cluster and also when removing one). > > When you add a node the cluster starts to rebalance. The new node, once > shards are allocated, starts serving requests. Since we have a cluster with > constant indexing and queries, this causes an IO bottleneck on the new > node. We're on EC2, not using provisioned IOps - I know that can help but > wondering if there is some cluster configuration that might ease the load > on cluster scaling. > > One thing I'm trying is reducing the cluster\_concurrent\_rebalance setting > to 1 to reduce moving shards. > There's also the throttling setting but I'm afraid that might impact > overall performance. > Any other recommendations on how to handle cluster scaling on a live > cluster? > > The ideal situation was to be able to tell the cluster to copy shards to > the new node without transferring traffic and removing old shards yet, and > make the new node active only when rebalancing has finished. I guess that's > not a possibility right now... -- ---
**Author:** ![rore](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rore/32/399_2.png) [@rore](https://discuss.elastic.co/u/rore)\ **Post date:** [January 22, 2013, 6:38pm UTC](https://discuss.elastic.co/t/cluster-performance-when-adding-a-node/10442/3 "2013-01-22T18:38:31Z")
Thanks. Will try that. On Jan 21, 2013 10:14 PM, "simonw" [simon.willnauer@elasticsearch.com](mailto:simon.willnauer@elasticsearch.com) wrote: > Hey, > > I would recommend you to throttle recovery IO during adding a node to the > cluster with something like > > "indices.recovery.max\_size\_\*\*per\_sec" : 20mb > > it might also make sense to set > "indices.recovery.concurrent\_\*\*streams" : 1 > > simon > > On Monday, January 21, 2013 7:42:10 PM UTC+1, Rotem wrote: > > > We encounter a lot of problems with cluster performance when trying to > > scale it (adding a node to the cluster and also when removing one). > > > > When you add a node the cluster starts to rebalance. The new node, once > > shards are allocated, starts serving requests. Since we have a cluster with > > constant indexing and queries, this causes an IO bottleneck on the new > > node. We're on EC2, not using provisioned IOps - I know that can help but > > wondering if there is some cluster configuration that might ease the load > > on cluster scaling. > > > > One thing I'm trying is reducing the cluster\_concurrent\_rebalancesetting to 1 to reduce moving shards. > > There's also the throttling setting but I'm afraid that might impact > > overall performance. > > Any other recommendations on how to handle cluster scaling on a live > > cluster? > > > > The ideal situation was to be able to tell the cluster to copy shards to > > the new node without transferring traffic and removing old shards yet, and > > make the new node active only when rebalancing has finished. I guess that's > > not a possibility right now... > > > > -- -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:55am UTC](https://discuss.elastic.co/t/cluster-performance-when-adding-a-node/10442/4 "2017-07-06T02:55:10Z")