Index routing allocation require does not work and throws confusing explain

[leandrojmp (Leandro Pereira)]

Hello,

I have a cluster on version 7.17 where I'm using a node attribute called node_type with the values hot and warm to allocate shards and move between node type with ILM, all indices have an index.routing.allocation.require.node_type: "hot" setting in the template and this work as expected.

Recently I'm starting to create a new cluster on 8.4 and decided to use the native data_tier role to allocate shards and move between the nodes with ILM, I have nodes that only have the role data_hot and notes that have the role data_warm and data_content.

The documentation says that _tier is a supported attribute for the index.routing.allocation.require.{attribute} setting, so I used the following setting in my template.

index.routing.allocation.require._tier: "data_hot"

But when I try to creat an index, the cluster went red because it was not able to allocate any shard, using the allocation explain API I got the following response.

    {
      "node_id": "QUc9-REDACTED",
      "node_name": "HOT-NODE",
      "transport_address": "REDACTED:9300",
      "node_attributes": {
        "xpack.installed": "true"
      },
      "node_decision": "no",
      "weight_ranking": 1,
      "deciders": [
        {
          "decider": "filter",
          "decision": "NO",
          "explanation": """node does not match index setting [index.routing.allocation.require] filters [_tier:"data_hot"]"""
        },
        {
          "decider": "data_tier",
          "decision": "NO",
          "explanation": "index has a preference for tiers [data_content] and node does not meet the required [data_content] tier"
        }
      ]
    }

It says that the node with the data_hot role does not match the setting _tier: data_hot, which is not true.

    "QUc9-REDACTED": {
      "name": "HOT-NODE",
      "transport_address": "REDACTED:9300",
      "host": "REDACTED",
      "ip": "REDACTED",
      "version": "8.4.2",
      "build_flavor": "default",
      "build_type": "rpm",
      "build_hash": "89f8c6d8429db93b816403ee75e5c270b43a940a",
      "total_indexing_buffer": 3221225472,
      "roles": [
        "data_hot",
        "ingest",
        "remote_cluster_client"
      ],
      "attributes": {
        "xpack.installed": "true"
      }

It also has a _tier_preference for a data_content role node, whic the hot node does not have, but the warm node has this role.

In the warm node the allocation explain is the following:

    {
      "node_id": "IElT-REDACTED",
      "node_name": "WARM-NODE",
      "transport_address": "REDACTED:9300",
      "node_attributes": {
        "xpack.installed": "true"
      },
      "node_decision": "no",
      "weight_ranking": 4,
      "deciders": [
        {
          "decider": "filter",
          "decision": "NO",
          "explanation": """node does not match index setting [index.routing.allocation.require] filters [_tier:"data_hot"]"""
        }
      ]
    }

If I add the setting index.routing.allocation.include._tier_preference: "data_hot" the shard will then be allocated to the hot nodes.

But I want this to work with the require setting as it work when using a custom attribute.

What is the issue here? Is this a bug? Is the documentation wrong and the _tier is not supported?