# BUG: Elasticsearch ignoring node.roles after upgrade from 7.6.1 **URL:** \ **Category:** Elasticsearch\ **Tags:** ilm-index-lifecycle-management\ **Created:** [May 31, 2021, 7:47am UTC](https://discuss.elastic.co/t/bug-elasticsearch-ignoring-node-roles-after-upgrade-from-7-6-1/274465 "2021-05-31T07:47:11Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![mverbeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mverbeek/32/101709_2.png) [@mverbeek](https://discuss.elastic.co/u/mverbeek)\ **Post date:** [May 31, 2021, 7:47am UTC](https://discuss.elastic.co/t/bug-elasticsearch-ignoring-node-roles-after-upgrade-from-7-6-1/274465/1 "2021-05-31T07:47:11Z")
Hello everyone, When upgrading from 7.6.1 to 7.10.2, I replaced the old node.master and node.data setting to the new node.roles setting. I changed 2 of the 5 data nodes to data\_cold only. But if the node has already been used as an data node before it will not respect this role. It will start and work as expected but it will still hold warm/hot shards. You can also still move warm/hot shards to the node. When I do an fresh install with 7.10.2 with the same config and cluster settings it does use the data\_cold role as expected. I typed out an simple log to recreate this problem if anyone wants to recreate it. Note: upgrading to the newest version of Elastic also did not work. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 31, 2021, 7:48am UTC](https://discuss.elastic.co/t/bug-elasticsearch-ignoring-node-roles-after-upgrade-from-7-6-1/274465/2 "2021-05-31T07:48:33Z")
Can you share your config please. ---
**Author:** ![mverbeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mverbeek/32/101709_2.png) [@mverbeek](https://discuss.elastic.co/u/mverbeek)\ **Post date:** [May 31, 2021, 7:51am UTC](https://discuss.elastic.co/t/bug-elasticsearch-ignoring-node-roles-after-upgrade-from-7-6-1/274465/3 "2021-05-31T07:51:48Z")
```auto cluster.name: test-cluster node.name: elkm02 path.data: /var/lib/elasticsearch path.logs: /var/log/elasticsearch network.host: 192.168.1.5 discovery.seed_hosts: ["192.168.1.1", "192.168.1.2", "192.168.1.3", "192.168.1.5"] cluster.initial_master_nodes: ["192.168.1.1", "192.168.1.5"] node.roles: master, ingest ``` Ofcourse, this is the config of the cluster I manged to recreate the problem in. This is the second test master, only changes are the name and roles ---
**Author:** ![mverbeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mverbeek/32/101709_2.png) [@mverbeek](https://discuss.elastic.co/u/mverbeek)\ **Post date:** [May 31, 2021, 11:31am UTC](https://discuss.elastic.co/t/bug-elasticsearch-ignoring-node-roles-after-upgrade-from-7-6-1/274465/4 "2021-05-31T11:31:58Z")
Any luck finding something? I can post the steps for what I did to recreate the problem if that helps. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [May 31, 2021, 7:14pm UTC](https://discuss.elastic.co/t/bug-elasticsearch-ignoring-node-roles-after-upgrade-from-7-6-1/274465/5 "2021-05-31T19:14:14Z")
Could you share `GET _cat/nodes` from the cluster exhibiting the problem? If after completing the upgrade you do a further rolling restart (i.e. restart all nodes, one-by-one) does the problem persist? ---
**Author:** ![mverbeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mverbeek/32/101709_2.png) [@mverbeek](https://discuss.elastic.co/u/mverbeek)\ **Post date:** [June 1, 2021, 9:19am UTC](https://discuss.elastic.co/t/bug-elasticsearch-ignoring-node-roles-after-upgrade-from-7-6-1/274465/6 "2021-06-01T09:19:15Z")
```auto 192.168.1.3 56 95 4 0.33 0.14 0.13 hsw - elkd02 192.168.1.4 8 95 5 0.16 0.03 0.05 c - elkd03 192.168.1.5 34 95 2 0.08 0.02 0.03 im * elkm02 192.168.1.2 15 95 1 0.62 0.86 0.87 hsw - elkd01 192.168.1.1 46 94 2 0.29 0.25 0.22 im - elkm01 ``` Yes, even if I put the entire cluster down and up the problem still persists. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [June 1, 2021, 9:34am UTC](https://discuss.elastic.co/t/bug-elasticsearch-ignoring-node-roles-after-upgrade-from-7-6-1/274465/7 "2021-06-01T09:34:11Z")
Ok would you use the [cluster allocation explain API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-allocation-explain.html) to explain the allocation of one of the shards you think to be allocated in the wrong place? ---
**Author:** ![mverbeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mverbeek/32/101709_2.png) [@mverbeek](https://discuss.elastic.co/u/mverbeek)\ **Post date:** [June 1, 2021, 9:53am UTC](https://discuss.elastic.co/t/bug-elasticsearch-ignoring-node-roles-after-upgrade-from-7-6-1/274465/8 "2021-06-01T09:53:05Z")
```auto { "error" : { "root_cause" : [ { "type" : "illegal_argument_exception", "reason" : "unable to find any unassigned shards to explain [ClusterAllocationExplainRequest[useAnyUnassignedShard=true,includeYesDecisions?=false]" } ], "type" : "illegal_argument_exception", "reason" : "unable to find any unassigned shards to explain [ClusterAllocationExplainRequest[useAnyUnassignedShard=true,includeYesDecisions?=false]" }, "status" : 400 } ``` That is the weird thing. The server thinks that its all fine, even tho there are hot/warm indices on the cold node ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [June 1, 2021, 10:23am UTC](https://discuss.elastic.co/t/bug-elasticsearch-ignoring-node-roles-after-upgrade-from-7-6-1/274465/9 "2021-06-01T10:23:33Z")
You need to tell the API which shard to explain, otherwise it just picks a random unassigned one and fails if all shards are assigned. ---
**Author:** ![mverbeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mverbeek/32/101709_2.png) [@mverbeek](https://discuss.elastic.co/u/mverbeek)\ **Post date:** [June 1, 2021, 11:08am UTC](https://discuss.elastic.co/t/bug-elasticsearch-ignoring-node-roles-after-upgrade-from-7-6-1/274465/10 "2021-06-01T11:08:22Z")
Sorry, my bad. Here it is: ```auto { "index" : "my-index-000006", "shard" : 0, "primary" : true, "current_state" : "started", "current_node" : { "id" : "cFGH4_FKRoKPlEg8rPD6Mg", "name" : "elkd03", "transport_address" : "192.168.1.4:9300", "attributes" : { "xpack.installed" : "true", "transform.node" : "false" }, "weight_ranking" : 1 }, "can_remain_on_current_node" : "yes", "can_rebalance_cluster" : "yes", "can_rebalance_to_other_node" : "no", "rebalance_explanation" : "cannot rebalance as no target node exists that can both allocate this shard and improve the cluster balance", "node_allocation_decisions" : [ { "node_id" : "PEb7VU_1RKa8q3HN8J7LCA", "node_name" : "elkd02", "transport_address" : "192.168.1.3:9300", "node_attributes" : { "xpack.installed" : "true", "transform.node" : "false" }, "node_decision" : "worse_balance", "weight_ranking" : 1 }, { "node_id" : "iNdDsFRiSquDN_V8iXwSPA", "node_name" : "elkd01", "transport_address" : "192.168.1.2:9300", "node_attributes" : { "xpack.installed" : "true", "transform.node" : "false" }, "node_decision" : "worse_balance", "weight_ranking" : 1 } ] } ``` This is an newly made index with one shard that is located on the cold node. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [June 1, 2021, 11:28am UTC](https://discuss.elastic.co/t/bug-elasticsearch-ignoring-node-roles-after-upgrade-from-7-6-1/274465/11 "2021-06-01T11:28:57Z")
> [@mverbeek](#): > > `"can_remain_on_current_node" : "yes",` Ok this shard can be allocated to all three nodes. What does `GET /my-index-000006/_settings` return? What steps have you taken to exclude it from the cold node? ---
**Author:** ![mverbeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mverbeek/32/101709_2.png) [@mverbeek](https://discuss.elastic.co/u/mverbeek)\ **Post date:** [June 1, 2021, 11:36am UTC](https://discuss.elastic.co/t/bug-elasticsearch-ignoring-node-roles-after-upgrade-from-7-6-1/274465/12 "2021-06-01T11:36:03Z")
> [@DavidTurner](#): > > `GET /my-index-000006/_settings` ```auto { "my-index-000006" : { "settings" : { "index" : { "creation_date" : "1622533255026", "number_of_shards" : "1", "number_of_replicas" : "0", "uuid" : "aTg9qJNSQxOEJwl-IVU1wA", "version" : { "created" : "7060199", "upgraded" : "7100299" }, "provided_name" : "my-index-000006" } } } } ``` Currently on this test environment nothing outside of the roles, If I recreate it like this with an fresh install the shard cannot be moved to that cold node since it does not meet the requirement. ```auto [NO(index has a preference for tiers [data_content] and node does not meet the required [data_content] tier)] ``` ---
**Author:** ![mverbeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mverbeek/32/101709_2.png) [@mverbeek](https://discuss.elastic.co/u/mverbeek)\ **Post date:** [June 3, 2021, 6:43am UTC](https://discuss.elastic.co/t/bug-elasticsearch-ignoring-node-roles-after-upgrade-from-7-6-1/274465/13 "2021-06-03T06:43:40Z")
Would it be possible that Elastic has an internal system that prevents the cold/hot/warm role from working until there are enough managed indices? I have been testing with that theory and it seems to be working but then why do the roles work immediately with an fresh install. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [June 3, 2021, 7:01am UTC](https://discuss.elastic.co/t/bug-elasticsearch-ignoring-node-roles-after-upgrade-from-7-6-1/274465/14 "2021-06-03T07:01:40Z")
This index has no settings to restrict its allocation to any particular tier so it can be allocated anywhere. If you check `GET /$INDEX/_settings` on your fresh install you will see that newer indices do have allocation settings applied. If you want to restrict the older indices to particular tiers you'll need to apply those settings yourself. ---
**Author:** ![mverbeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mverbeek/32/101709_2.png) [@mverbeek](https://discuss.elastic.co/u/mverbeek)\ **Post date:** [June 3, 2021, 7:08am UTC](https://discuss.elastic.co/t/bug-elasticsearch-ignoring-node-roles-after-upgrade-from-7-6-1/274465/15 "2021-06-03T07:08:03Z")
Weird then, then I still can't explain the behavior on the production server. It ignored the roles completely until I added an new cold policy to change an bunch of older indices to cold. But we already had an working hot/warm/cold ILM in place at that time. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 1, 2021, 7:08am UTC](https://discuss.elastic.co/t/bug-elasticsearch-ignoring-node-roles-after-upgrade-from-7-6-1/274465/16 "2021-07-01T07:08:15Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.