# Controlled rotation of elasticsearch data nodes while enabling the shard allocation awareness **URL:** \ **Category:** Elasticsearch\ **Created:** [July 13, 2023, 2:43am UTC](https://discuss.elastic.co/t/controlled-rotation-of-elasticsearch-data-nodes-while-enabling-the-shard-allocation-awareness/338269 "2023-07-13T02:43:04Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![veerachenna](https://avatars.discourse-cdn.com/v4/letter/v/13edae/32.png) [@veerachenna](https://discuss.elastic.co/u/veerachenna)\ **Post date:** [July 13, 2023, 2:43am UTC](https://discuss.elastic.co/t/controlled-rotation-of-elasticsearch-data-nodes-while-enabling-the-shard-allocation-awareness/338269/1 "2023-07-13T02:43:04Z")
Hi All, We are trying to enable the shard allocation awareness on the elasticsearch cluster on "zone" attribute while rotating the data nodes one after the other. We wanted to achieve this in more controlled manner. Initial cluster state is that none of the data nodes have any value set for the attribute "zone". To do that following are the steps followed, 1. Enable the shard allocation awareness on the master nodes by setting ```auto cluster.routing.allocation.awareness.attributes: zone ``` 1. Disable shard allocation on the cluster using the following ```auto curl -X PUT "localhost:9200/_cluster/settings?pretty" -H 'Content-Type: application/json' -d' { "persistent": { "cluster.routing.allocation.enable": "none" } } ' ``` 1. Boot new data nodes with the "zone" attribute. 2. Exclude one of the old data node to move the shards from that node to new data nodes with "zone" attribute. ```auto curl -X PUT "localhost:9200/_cluster/settings?pretty" -H 'Content-Type: application/json' -d' { "transient": { "cluster.routing.allocation.exclude._ip": "x.y.y.y" } } ' ``` But this is not working. None of the shards from the excluded old data node are moving to the new data nodes. The only option we can see is to move shard one by one from the old data node using the "\_cluster/reroute" API. But this feels wrong. Also if we enable the shard allocation by setting "cluster.routing.allocation.enable" to "all", all the shards from the old nodes will start moving to the new data nodes at once which might cause cluster to become unstable. Is there any way to achieve controlled rollout? PS: We cannot restart nodes by adding attribute. The reason is we wanted to do few more changes to the nodes at the infra level. Thanks in advance ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [July 13, 2023, 3:08am UTC](https://discuss.elastic.co/t/controlled-rotation-of-elasticsearch-data-nodes-while-enabling-the-shard-allocation-awareness/338269/2 "2023-07-13T03:08:20Z")
> [@veerachenna](#): > > `"cluster.routing.allocation.enable": "none"` This disables shard allocations, so it is expected that the shards will not move to the new nodes. > No shard allocations of any kind are allowed for any indices. You need to have it set to `all` or at least `primaries` so the shards of current indices can be allocated in the new nodes. I do not entirely get what you are trying to do. Are you just swapping nodes? Removing some old nodes and adding new nodes? ---
**Author:** ![veerachenna](https://avatars.discourse-cdn.com/v4/letter/v/13edae/32.png) [@veerachenna](https://discuss.elastic.co/u/veerachenna)\ **Post date:** [July 13, 2023, 4:32am UTC](https://discuss.elastic.co/t/controlled-rotation-of-elasticsearch-data-nodes-while-enabling-the-shard-allocation-awareness/338269/3 "2023-07-13T04:32:09Z")
Yes @leandrojmp . We are trying to swap the nodes along with enabling shard allocation awareness. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 13, 2023, 6:03am UTC](https://discuss.elastic.co/t/controlled-rotation-of-elasticsearch-data-nodes-while-enabling-the-shard-allocation-awareness/338269/4 "2023-07-13T06:03:17Z")
> [@veerachenna](#): > > all the shards from the old nodes will start moving to the new data nodes at once which might cause cluster to become unstable. By default Elasticsearch will only move up to two shards at once, to avoid any such instability. ---
**Author:** ![veerachenna](https://avatars.discourse-cdn.com/v4/letter/v/13edae/32.png) [@veerachenna](https://discuss.elastic.co/u/veerachenna)\ **Post date:** [July 13, 2023, 10:03am UTC](https://discuss.elastic.co/t/controlled-rotation-of-elasticsearch-data-nodes-while-enabling-the-shard-allocation-awareness/338269/5 "2023-07-13T10:03:36Z")
@DavidTurner , Are you talking about which of these two properties? ```auto cluster.routing.allocation.node_concurrent_recoveries cluster.routing.allocation.cluster_concurrent_rebalance ``` Also if we want to stop the recoveries/rebalance during the peak traffic hours, can we make it to zero? Will that solve my problem of controlled rollout? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 13, 2023, 10:19am UTC](https://discuss.elastic.co/t/controlled-rotation-of-elasticsearch-data-nodes-while-enabling-the-shard-allocation-awareness/338269/6 "2023-07-13T10:19:23Z")
> [@veerachenna](#): > > Are you talking about which of these two properties? Those two settings are related, yes, but I strongly recommend you leave them at their default values always. > [@veerachenna](#): > > Also if we want to stop the recoveries/rebalance during the peak traffic hours, can we make it to zero? Will that solve my problem of controlled rollout? If your cluster is properly configured then there should be no need to avoid recoveries during peak traffic hours. "Properly configured" here includes leaving the settings you mentioned above at their default values. ---
**Author:** ![veerachenna](https://avatars.discourse-cdn.com/v4/letter/v/13edae/32.png) [@veerachenna](https://discuss.elastic.co/u/veerachenna)\ **Post date:** [July 13, 2023, 10:31am UTC](https://discuss.elastic.co/t/controlled-rotation-of-elasticsearch-data-nodes-while-enabling-the-shard-allocation-awareness/338269/7 "2023-07-13T10:31:03Z")
Thanks @DavidTurner for the reply. Have one more doubt to control the speed of the recovery. Which of the following two properties would help to control the speed of recovery, ```auto indices.store.throttle.max_bytes_per_sec (since we are on 5.6) indices.recovery.max_bytes_per_sec ``` ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 13, 2023, 10:44am UTC](https://discuss.elastic.co/t/controlled-rotation-of-elasticsearch-data-nodes-while-enabling-the-shard-allocation-awareness/338269/8 "2023-07-13T10:44:16Z")
Oh, sorry, the responses I gave above assumed you were on a supported & maintained version (7.17 or later). 5.6 passed EOL over 5 years ago and I don't remember how things worked back then. You need to upgrade as a matter of urgency. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 10, 2023, 10:44am UTC](https://discuss.elastic.co/t/controlled-rotation-of-elasticsearch-data-nodes-while-enabling-the-shard-allocation-awareness/338269/9 "2023-08-10T10:44:28Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.