Shard activity always happening in ElasticSearch 8.12 version

Elastic Stack Elasticsearch
1

We have upgraded Elasticsearch to 8.12 and changed index level settings to have shards per node as 4 , which was earlier 2.
Somehow everytime some shard recovery is happening and shards are moving from one node to another. Cluster is in Green status only.
{
"persistent": {
"cluster": {
"routing": {
"allocation": {
"node_concurrent_recoveries": "6",
"exclude": {
"_host": "atl1s11cjbesd36.xt.local,"
}
}
}
},
"indices": {
"breaker": {
"total": {
"limit": "80%"
}
},
"recovery": {
"max_bytes_per_sec": "200mb",
"max_concurrent_file_chunks": "4"
}
},
"logger": {
"org": {
"elasticsearch": "WARN"
}
}
},
"transient": {
"cluster": {
"routing": {
"allocation": {
"node_concurrent_recoveries": "8"
}
}
}
}
}

2

See the docs for this setting:

Increasing this setting may cause shard movements to have a performance impact on other activity in your cluster, but may not make shard movements complete noticeably sooner. We do not recommend adjusting this setting from its default of 2 .

If you've only just upgraded then you will need to allow some time for your cluster to adjust itself to the new balancing heuristics. Depending on cluster size this could be several days of work (or more).

See also this troubleshooting guide.

3

Thanks @DavidTurner !
I ran this API GET _internal/desired_balance and i got below response for cluster.
“node_is_desired” false - 3322
“node_is_desired” true - 6684
Does it look OK?
Also. is this rebalance logic introduced in 8.6 going to impact cluster performance in big way or there has been some benchmarking on this?

4

I expect that to improve over time.

Yes generally it will improve performance.

5

Thanks @DavidTurner !
Is there any metric change which suggests things are stabilising now?
Like change in “node_is_desired” metric , will “node_is_desired” false -decrease over time?

6

Yes the node_is_desired flags should all end up true.

7

Thanks @DavidTurner !
I see that shard recoveries after version upgrade impacts indexing rate, esp when bulk indexing is being done which goes to shards on different nodes. Will "cluster.routing.allocation.balance.threshold" help in this case?

8

Probably, but I think the more likely problem is that you have set indices.recovery.max_bytes_per_sec: 200mb but your hardware can't cope with that. See these docs for more info:

If this limit is too high, ongoing recoveries may consume an excess of bandwidth and other resources, which can have a performance impact on your cluster and in extreme cases may destabilize it.