# Shard recovery in 8.10.2 seems to happen more often

**URL:** <https://discuss.elastic.co/t/shard-recovery-in-8-10-2-seems-to-happen-more-often/357024>\
**Category:** Elasticsearch\
**Created:** [April 8, 2024, 10:53pm UTC](https://discuss.elastic.co/t/shard-recovery-in-8-10-2-seems-to-happen-more-often/357024 "2024-04-08T22:53:11Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![linkerc](https://avatars.discourse-cdn.com/v4/letter/l/13edae/32.png) [@linkerc](https://discuss.elastic.co/u/linkerc)\
**Post date:** [April 8, 2024, 10:53pm UTC](https://discuss.elastic.co/t/shard-recovery-in-8-10-2-seems-to-happen-more-often/357024/1 "2024-04-08T22:53:11Z")

</div>

I have noticed that in version 8.10.2, shard allocation seems to prioritize on storage space first then the shard delta.  
Which is fine. I don't really have a preference either way.  
But as the result, shard moving seems to be happening more often now.  
By default, the shard count delta between data nodes are larger in version 8.10.2.  
So I constantly see recovery tasks running.  
This is not the case for version 7.15. Older version is more strict on maintaining shard count difference. So I can set "cluster.routing.allocation.balance.threshold" to say 3 and the delta will remain within 3 between data nodes. I rarely see shard recovery (unless I delete indices).

But with version 8.10.2, I always see recovery. Is this expected?

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [April 9, 2024, 4:18am UTC](https://discuss.elastic.co/t/shard-recovery-in-8-10-2-seems-to-happen-more-often/357024/2 "2024-04-09T04:18:50Z")

</div>

> [@linkerc](#):
>
> But with version 8.10.2, I always see recovery. Is this expected?

Which version were you running before?

Did you change any of the default settings regarding rebalance?

- `cluster.routing.allocation.cluster_concurrent_rebalance`
- `cluster.routing.allocation.node_concurrent_incoming_recoveries`
- `cluster.routing.allocation.node_concurrent_outgoing_recoveries`
- `cluster.routing.allocation.node_concurrent_recoveries`

---

<div class="post-metadata">

**Author:** ![linkerc](https://avatars.discourse-cdn.com/v4/letter/l/13edae/32.png) [@linkerc](https://discuss.elastic.co/u/linkerc)\
**Post date:** [April 9, 2024, 5:55pm UTC](https://discuss.elastic.co/t/shard-recovery-in-8-10-2-seems-to-happen-more-often/357024/3 "2024-04-09T17:55:26Z")

</div>

7.15  
For 7.15, we change the following:

```auto
PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.balance.threshold": 3,
    "cluster.routing.allocation.cluster_concurrent_rebalance": null,
    "cluster.routing.allocation.node_concurrent_recoveries": null,
    "cluster.routing.allocation.node_concurrent_incoming_recoveries":null,
    "cluster.routing.allocation.node_concurrent_outgoing_recoveries":null
  }
}

```

So shard delta between data nodes stayed typically \<= 3.

---

<div class="post-metadata">

**Author:** ![linkerc](https://avatars.discourse-cdn.com/v4/letter/l/13edae/32.png) [@linkerc](https://discuss.elastic.co/u/linkerc)\
**Post date:** [April 9, 2024, 8:21pm UTC](https://discuss.elastic.co/t/shard-recovery-in-8-10-2-seems-to-happen-more-often/357024/4 "2024-04-09T20:21:08Z")

</div>

Version 8.10.2 seems to be in a perpetual cycle of shard rebalancing.  
After setting "cluster.routing.allocation.cluster\_concurrent\_rebalance" to 30 and the rebalancing jobs jumped from 2 to 30 immediately.

I'll keep it at 30 to see if it's simply too many for the default of 2 to catch up...

But fundamentally seems to be the issue of too many shard movements. The cluster doesn't settle to a state.  
We do create/delete some indices hourly & daily, but version 7.15 suggests our pattern of usage was not an issue for older algorithm.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [April 9, 2024, 8:43pm UTC](https://discuss.elastic.co/t/shard-recovery-in-8-10-2-seems-to-happen-more-often/357024/5 "2024-04-09T20:43:25Z")

</div>

> [@linkerc](#):
>
> I'll keep it at 30 to see if it's simply too many for the default of 2 to catch up.

You should use the default, there is an open issue about this, increasing this value can impact in the rebalance.

Check this issue: [Increasing `cluster.routing.allocation.cluster_concurrent_rebalance` causes redundant shard movements · Issue #87279 · elastic/elasticsearch · GitHub](https://github.com/elastic/elasticsearch/issues/87279)

And also this one tracking a possible fix: [Throttle recoveries on data nodes instead of master · Issue #98087 · elastic/elasticsearch · GitHub](https://github.com/elastic/elasticsearch/issues/98087)

> [@linkerc](#):
>
> We do create/delete some indices hourly & daily, but version 7.15 suggests our pattern of usage was not an issue for older algorithm.

There was also a change on version 8.6 that changed some things regarding balancing.

I had some issues when I upgraded to 8.8.1 about shards that kept moving around, as mentioned in this [topic](https://discuss.elastic.co/t/cluster-shards-unbalanced-and-keep-moving-shards-around-after-upgrade-to-8-8-1/336573).

---

<div class="post-metadata">

**Author:** ![linkerc](https://avatars.discourse-cdn.com/v4/letter/l/13edae/32.png) [@linkerc](https://discuss.elastic.co/u/linkerc)\
**Post date:** [April 9, 2024, 8:54pm UTC](https://discuss.elastic.co/t/shard-recovery-in-8-10-2-seems-to-happen-more-often/357024/6 "2024-04-09T20:54:38Z")

</div>

Ok. Thanks.

---

<div class="post-metadata">

**Author:** ![linkerc](https://avatars.discourse-cdn.com/v4/letter/l/13edae/32.png) [@linkerc](https://discuss.elastic.co/u/linkerc)\
**Post date:** [April 9, 2024, 9:07pm UTC](https://discuss.elastic.co/t/shard-recovery-in-8-10-2-seems-to-happen-more-often/357024/7 "2024-04-09T21:07:50Z")

</div>

I think part of the reason contributing to what I'm seeing is the hourly index creation/deletion in our use case with the new algo.

GET \_internal/desired\_balance  
all of a sudden shows 10 nodes to be false on the hour.

I guess the new introduction to balancing primary on storage rendered shard count unimportant, yet it triggers rebalance.  
It becomes too jittery..

---

<div class="post-metadata">

**Author:** ![linkerc](https://avatars.discourse-cdn.com/v4/letter/l/13edae/32.png) [@linkerc](https://discuss.elastic.co/u/linkerc)\
**Post date:** [April 9, 2024, 10:23pm UTC](https://discuss.elastic.co/t/shard-recovery-in-8-10-2-seems-to-happen-more-often/357024/8 "2024-04-09T22:23:58Z")

</div>

After reset all cluster settings to default. I just got 40 "node\_is\_desired": false.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [April 10, 2024, 6:57am UTC](https://discuss.elastic.co/t/shard-recovery-in-8-10-2-seems-to-happen-more-often/357024/9 "2024-04-10T06:57:33Z")

</div>

> [@linkerc](#):
>
> It becomes too jittery..

This typically means you should increase `cluster.routing.allocation.balance.threshold`. Looks like you currently have it set at `3`, but you can reasonably go much higher than that.

---

<div class="post-metadata">

**Author:** ![linkerc](https://avatars.discourse-cdn.com/v4/letter/l/13edae/32.png) [@linkerc](https://discuss.elastic.co/u/linkerc)\
**Post date:** [April 10, 2024, 7:55pm UTC](https://discuss.elastic.co/t/shard-recovery-in-8-10-2-seems-to-happen-more-often/357024/10 "2024-04-10T19:55:44Z")

</div>

ok. I'll give that a try.  
Just to mention that if I set it to 3, it's never settle below delta of 3. I attributed to prioritizing on storage with 8.10.2.  
Even with delta above the threshold, I do see the cluster settle for period of times. Meaning no rebalancing occuring, which I interpreted that threshold is no longer useful.

I'll give it a larger value to see if it's less jittery.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [April 10, 2024, 8:23pm UTC](https://discuss.elastic.co/t/shard-recovery-in-8-10-2-seems-to-happen-more-often/357024/11 "2024-04-10T20:23:56Z")

</div>

Yeah `3` no longer means "shard counts balanced within ±3", it's balancing a mix of shard count, disk space and write load. That means one way to balance the cluster would be to put lots of tiny shards on one node and the few remaining enormous shards on the other node.

---

<div class="post-metadata">

**Author:** ![linkerc](https://avatars.discourse-cdn.com/v4/letter/l/13edae/32.png) [@linkerc](https://discuss.elastic.co/u/linkerc)\
**Post date:** [April 10, 2024, 8:45pm UTC](https://discuss.elastic.co/t/shard-recovery-in-8-10-2-seems-to-happen-more-often/357024/12 "2024-04-10T20:45:34Z")

</div>

thanks for the clarification.  
Does this mean the default of 1000 shards per node is no longer enforced?

And setting that value to 10 seems to settle the 8.10.2 cluster. Much less rebalancing.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [April 10, 2024, 9:21pm UTC](https://discuss.elastic.co/t/shard-recovery-in-8-10-2-seems-to-happen-more-often/357024/13 "2024-04-10T21:21:48Z")

</div>

> [@linkerc](#):
>
> Does this mean the default of 1000 shards per node is no longer enforced?

No, the average number of shards per (non-frozen) node must remain below 1000 by default.

---

<div class="post-metadata">

**Author:** ![mukularora89](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mukularora89/32/126696_2.png) [@mukularora89](https://discuss.elastic.co/u/mukularora89)\
**Post date:** [August 21, 2024, 3:51pm UTC](https://discuss.elastic.co/t/shard-recovery-in-8-10-2-seems-to-happen-more-often/357024/14 "2024-08-21T15:51:48Z")

</div>

Hi @linkerc what value did you set for cluster.routing.allocation.balance.threshold and it still does shard movements. Also does it keeping your cluster in unbalanced state?

---

<div class="post-metadata">

**Author:** ![linkerc](https://avatars.discourse-cdn.com/v4/letter/l/13edae/32.png) [@linkerc](https://discuss.elastic.co/u/linkerc)\
**Post date:** [January 16, 2025, 7:59pm UTC](https://discuss.elastic.co/t/shard-recovery-in-8-10-2-seems-to-happen-more-often/357024/15 "2025-01-16T19:59:11Z")

</div>

I use 3 - 5 now and it stops unnecessary shard movement. It works for our application so far.