Elastic ignoring cold_role

Elastic Stack Elasticsearch
1

We have an Elastic stack on 7.6.1 that we upgrades to 7.10.2. Then we introduced new nodes with slower bigger storage to it with cold_node role and set the others to data_warm, data_hot and data_content.
During the install an error was made and warm/hot shards where placed on the new cold_role nodes.
Now after everything is correct it still places shards from warm and hot indices on the cold nodes when balancing. I reproduced the error in an test environment and upgraded that environment to the latest version but the error/bug still remains.
So in the end we have en environment with the correct roles according to documentation but all the shards are still being balanced like they all have the same role.
If I do an correct install with this configuration without an upgrade or data involved it confirms that the configuration is correct and I am unable to move hot/warm index shards to the cold nodes.

Any ideas would be greatly appreciated.

2

Are you using ILM here?

3

Yes, but normally if you use the reroute command you can also not move to cold nodes if they are not cold and that is also still posibble

4

What is the output from GET /_cat/nodeattrs?

5

xpack.installed true
transform.node false/true (depending on node)
hypervisor "hypervisor-name"

Nothing about the cold roles

6

To add to this:
The _nodes/stats does list the correct roles

7

Do your index templates have any allocation config specified in them for when they are created?

8

Nope, it only checks that they are balanced across hypervisors. And those settings work in an fresh install while respecting the node role

9

What does your ILM policy and the matching templates look like?

10

De ILM policy is:

{
  "auditbeat-ilm" : {
    "version" : 2,
    "modified_date" : "2021-04-15T12:31:17.536Z",
    "policy" : {
      "phases" : {
        "warm" : {
          "min_age" : "1d",
          "actions" : {
            "forcemerge" : {
              "max_num_segments" : 1
            },
            "set_priority" : {
              "priority" : 50
            },
            "shrink" : {
              "number_of_shards" : 1
            }
          }
        },
        "cold" : {
          "min_age" : "93d",
          "actions" : {
            "allocate" : {
              "number_of_replicas" : 1,
              "include" : { },
              "exclude" : { },
              "require" : { }
            },
            "freeze" : { }
          }
        },
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "7d"
            },
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "delete" : {
          "min_age" : "367d",
          "actions" : {
            "delete" : {
              "delete_searchable_snapshot" : true
            }
          }
        }
      }
    }
  }
}

The template is the one provides by the modules, so in this ILM the auditbeat module

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.