# Shard activity always happening in ElasticSearch 8.12 version **URL:** \ **Category:** Elasticsearch\ **Created:** [August 14, 2024, 12:32pm UTC](https://discuss.elastic.co/t/shard-activity-always-happening-in-elasticsearch-8-12-version/364906 "2024-08-14T12:32:50Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Amit\_Shukla](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/amit_shukla/32/121862_2.png) [@Amit\_Shukla](https://discuss.elastic.co/u/Amit_Shukla)\ **Post date:** [August 14, 2024, 12:32pm UTC](https://discuss.elastic.co/t/shard-activity-always-happening-in-elasticsearch-8-12-version/364906/1 "2024-08-14T12:32:51Z")
We have upgraded Elasticsearch to 8.12 and changed index level settings to have shards per node as 4 , which was earlier 2. Somehow everytime some shard recovery is happening and shards are moving from one node to another. Cluster is in Green status only. { "persistent": { "cluster": { "routing": { "allocation": { "node\_concurrent\_recoveries": "6", "exclude": { "\_host": "atl1s11cjbesd36.xt.local," } } } }, "indices": { "breaker": { "total": { "limit": "80%" } }, "recovery": { "max\_bytes\_per\_sec": "200mb", "max\_concurrent\_file\_chunks": "4" } }, "logger": { "org": { "elasticsearch": "WARN" } } }, "transient": { "cluster": { "routing": { "allocation": { "node\_concurrent\_recoveries": "8" } } } } } ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [August 14, 2024, 1:17pm UTC](https://discuss.elastic.co/t/shard-activity-always-happening-in-elasticsearch-8-12-version/364906/2 "2024-08-14T13:17:57Z")
> [@Amit\_Shukla](#): > > node\_concurrent\_recoveries See [the docs for this setting](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cluster.html#cluster-shard-allocation-settings): > Increasing this setting may cause shard movements to have a performance impact on other activity in your cluster, but may not make shard movements complete noticeably sooner. We do not recommend adjusting this setting from its default of `2` . If you've only just upgraded then you will need to allow some time for your cluster to adjust itself to the new balancing heuristics. Depending on cluster size this could be several days of work (or more). See also [this troubleshooting guide](https://www.elastic.co/guide/en/elasticsearch/reference/current/troubleshooting-unbalanced-cluster.html). ---
**Author:** ![Amit\_Shukla](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/amit_shukla/32/121862_2.png) [@Amit\_Shukla](https://discuss.elastic.co/u/Amit_Shukla)\ **Post date:** [August 14, 2024, 3:38pm UTC](https://discuss.elastic.co/t/shard-activity-always-happening-in-elasticsearch-8-12-version/364906/3 "2024-08-14T15:38:16Z")
Thanks @DavidTurner ! I ran this API GET \_internal/desired\_balance and i got below response for cluster. “node\_is\_desired” false - 3322 “node\_is\_desired” true - 6684 Does it look OK? Also. is this rebalance logic introduced in 8.6 going to impact cluster performance in big way or there has been some benchmarking on this? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [August 14, 2024, 4:15pm UTC](https://discuss.elastic.co/t/shard-activity-always-happening-in-elasticsearch-8-12-version/364906/4 "2024-08-14T16:15:21Z")
> [@Amit\_Shukla](#): > > Does it look OK? I expect that to improve over time. > [@Amit\_Shukla](#): > > is this rebalance logic introduced in 8.6 going to impact cluster performance in big way or there has been some benchmarking on this? Yes generally it will improve performance. ---
**Author:** ![Amit\_Shukla](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/amit_shukla/32/121862_2.png) [@Amit\_Shukla](https://discuss.elastic.co/u/Amit_Shukla)\ **Post date:** [August 15, 2024, 5:43am UTC](https://discuss.elastic.co/t/shard-activity-always-happening-in-elasticsearch-8-12-version/364906/5 "2024-08-15T05:43:01Z")
> [@Amit\_Shukla](#): > > “node\_is\_desired” Thanks @DavidTurner ! Is there any metric change which suggests things are stabilising now? Like change in “node\_is\_desired” metric , will “node\_is\_desired” false -decrease over time? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [August 15, 2024, 6:27am UTC](https://discuss.elastic.co/t/shard-activity-always-happening-in-elasticsearch-8-12-version/364906/6 "2024-08-15T06:27:41Z")
> [@Amit\_Shukla](#): > > Is there any metric change which suggests things are stabilising now? Yes the `node_is_desired` flags should all end up `true`. ---
**Author:** ![Amit\_Shukla](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/amit_shukla/32/121862_2.png) [@Amit\_Shukla](https://discuss.elastic.co/u/Amit_Shukla)\ **Post date:** [August 23, 2024, 2:28pm UTC](https://discuss.elastic.co/t/shard-activity-always-happening-in-elasticsearch-8-12-version/364906/7 "2024-08-23T14:28:17Z")
Thanks @DavidTurner ! I see that shard recoveries after version upgrade impacts indexing rate, esp when bulk indexing is being done which goes to shards on different nodes. Will "cluster.routing.allocation.balance.threshold" help in this case? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [August 23, 2024, 3:54pm UTC](https://discuss.elastic.co/t/shard-activity-always-happening-in-elasticsearch-8-12-version/364906/8 "2024-08-23T15:54:46Z")
> [@Amit\_Shukla](#): > > Will "cluster.routing.allocation.balance.threshold" help in this case? Probably, but I think the more likely problem is that you have set `indices.recovery.max_bytes_per_sec: 200mb` but your hardware can't cope with that. See [these docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/recovery.html#recovery-settings) for more info: > If this limit is too high, ongoing recoveries may consume an excess of bandwidth and other resources, which can have a performance impact on your cluster and in extreme cases may destabilize it.