Cluster rebalancing

Elastic Stack Elasticsearch
1

I have an issue with ES automatic rebalancing shards in my cluster:

  • I’m using Elasticsearch 7.17.9
  • I have a cluster with 17 data nodes that all of them have data role
  • Data nodes already have cluster.routing.allocation.awareness.attributes : group
    • 12 data nodes set to group hot and 5 of them set to group cold
  • Shard allocation settings are in their default setting
  • I have 5 shards per index without replica
  • I’m using Curator to transition indices between the hot and cold nodes

The shard allocation is as follows:

shards  node
145    hot11
145    hot12
145    hot13
145    hot21
145    hot22
145    hot23
145    hot31
145    hot32
145    hot33
144    hot41
145    hot42
145    hot43
8125   cold1
8126   cold2
8125   cold3
8125   cold4
8125   cold5

I expect balancer works independently within each group (Hot and Cold), but it seems that Elastic wants to balance the entire cluster. It starts relocating shards from the cold2 to the hot nodes, but because it can’t match with the node group, it moves back to another cold node and this process is repeated in a loop.

As explain here

If you are using data tiers then Elasticsearch automatically applies allocation filtering rules to place each shard within the appropriate tier. These rules mean that the balancer works independently within each tier.

I changed my data node roles to data_hot and data_cold. However, the problem still persists.

2

Did you change the role of all your nodes and restarted them?

Do you have any allocation settings in your indices templates? Please share them.

3

yes, I changed all of them and then restarted

yes, sure

"settings": {
	"index": {
		"routing": {
			"allocation": {
				"include": {
					"group": "hot"
				}
			}
		}
	}
}

when an index is created, it is initially allocated to the hot group, and then the allocation is changed to cold by using Curator