Shard recovery in 8.10.2 seems to happen more often

[leandrojmp (Leandro Pereira)]

@linkerc

I'll keep it at 30 to see if it's simply too many for the default of 2 to catch up.

You should use the default, there is an open issue about this, increasing this value can impact in the rebalance.

Check this issue: Increasing `cluster.routing.allocation.cluster_concurrent_rebalance` causes redundant shard movements · Issue #87279 · elastic/elasticsearch · GitHub

And also this one tracking a possible fix: Throttle recoveries on data nodes instead of master · Issue #98087 · elastic/elasticsearch · GitHub

@linkerc

We do create/delete some indices hourly & daily, but version 7.15 suggests our pattern of usage was not an issue for older algorithm.

There was also a change on version 8.6 that changed some things regarding balancing.

I had some issues when I upgraded to 8.8.1 about shards that kept moving around, as mentioned in this topic.