# Shard numbers no longer equal (not even close) among cluster nodes **URL:** \ **Category:** Elasticsearch\ **Created:** [October 19, 2023, 2:24am UTC](https://discuss.elastic.co/t/shard-numbers-no-longer-equal-not-even-close-among-cluster-nodes/345342 "2023-10-19T02:24:21Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Hao\_Yellow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hao_yellow/32/102182_2.png) [@Hao\_Yellow](https://discuss.elastic.co/u/Hao_Yellow)\ **Post date:** [October 19, 2023, 2:24am UTC](https://discuss.elastic.co/t/shard-numbers-no-longer-equal-not-even-close-among-cluster-nodes/345342/1 "2023-10-19T02:24:21Z")
Hello, I've been recently upgraded an Elasticsearch cluster, with 5 nodes, from version 7.3 to 7.17 then 8.9. As always, I've never disabled shard allocation and rebalancing, so until 7.17 it's observed, and as I understand, that the shard numbers among the cluster nodes should be equal or close to equal if not possible to be exactly equal. But when I upgrade to 8.9, it seems that it's no longer that way. The 5 nodes have very different number of shards after automatic shard allocation is finished (by observing if there are any relocating shards, and checking `GET /_cat/recovery?active_only=true` which returns empty results) — The shard number of each node respectively: 112, 145, 132, 138, 142. The difference is not something like between 10 to 200, but still it's not even close to "equal". So what happens? Is it that Elasticsearch now has an improved strategy to allocate and rebalance shards among the cluster, so it's not necessary to be equal? I've also tried restarting the cluster, which didn't work. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [October 19, 2023, 3:15am UTC](https://discuss.elastic.co/t/shard-numbers-no-longer-equal-not-even-close-among-cluster-nodes/345342/2 "2023-10-19T03:15:45Z")
> [@Hao\_Yellow](#): > > So what happens? Is it that Elasticsearch now has an improved strategy to allocate and rebalance shards among the cluster, so it's not necessary to be equal? The Shard balancing heuristics was changed on version 8.6 to also consider the disk usage of the shards while rebalancing them. You should not expect your nodes to have an equal number of shards anymore. ---
**Author:** ![Hao\_Yellow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hao_yellow/32/102182_2.png) [@Hao\_Yellow](https://discuss.elastic.co/u/Hao_Yellow)\ **Post date:** [October 19, 2023, 3:35am UTC](https://discuss.elastic.co/t/shard-numbers-no-longer-equal-not-even-close-among-cluster-nodes/345342/3 "2023-10-19T03:35:45Z")
That's great! Thank you. According to [Shard rebalancing settings](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cluster.html#shards-rebalancing-settings): > A cluster is _balanced_ when it has an equal number of shards on each node, with all nodes needing equal resources, without having a concentration of shards from any index on any node. I'm not sure if it's a problem with my understanding of the language or something, but reading that part of documentation made me think that the cluster with such varying shard numbers (112, 145, 132, 138, 142) was far from _balanced_. But then as you pointed out, it feels like this way of balancing is more ideal, although one cannot realize if the cluster has finished rebalancing by looking at the shard numbers among all nodes. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [October 19, 2023, 9:26am UTC](https://discuss.elastic.co/t/shard-numbers-no-longer-equal-not-even-close-among-cluster-nodes/345342/4 "2023-10-19T09:26:48Z")
The key phrase (added to the docs in 8.6) is here: > [@Hao\_Yellow](#): > > with all nodes needing equal resources If your cluster contains shards with varying resource needs then Elasticsearch must find a compromise between equalizing the shard count and balancing the resources. ---
**Author:** ![Hao\_Yellow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hao_yellow/32/102182_2.png) [@Hao\_Yellow](https://discuss.elastic.co/u/Hao_Yellow)\ **Post date:** [October 19, 2023, 9:31am UTC](https://discuss.elastic.co/t/shard-numbers-no-longer-equal-not-even-close-among-cluster-nodes/345342/5 "2023-10-19T09:31:25Z")
That's quite reasonable. Thanks! ---
**Author:** ![sholzhauer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sholzhauer/32/110282_2.png) [@sholzhauer](https://discuss.elastic.co/u/sholzhauer)\ **Post date:** [October 19, 2023, 11:45am UTC](https://discuss.elastic.co/t/shard-numbers-no-longer-equal-not-even-close-among-cluster-nodes/345342/6 "2023-10-19T11:45:12Z")
Specifically have a look at [this portion](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cluster.html#shards-rebalancing-heuristics) of the docs you referenced: > The weight of a node depends on the number of shards it holds and on the total estimated resource usage of those shards expressed in terms of the size of the shard on disk and the number of threads needed to support write traffic to the shard. ---
**Author:** ![Hao\_Yellow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hao_yellow/32/102182_2.png) [@Hao\_Yellow](https://discuss.elastic.co/u/Hao_Yellow)\ **Post date:** [October 20, 2023, 1:58am UTC](https://discuss.elastic.co/t/shard-numbers-no-longer-equal-not-even-close-among-cluster-nodes/345342/7 "2023-10-20T01:58:34Z")
Great, thank you! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 17, 2023, 1:59am UTC](https://discuss.elastic.co/t/shard-numbers-no-longer-equal-not-even-close-among-cluster-nodes/345342/8 "2023-11-17T01:59:19Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.