Shards not relocating to warm nodes

[Smrithin_N_S]

I have a deployment with hot-warm architecture enabled. Below is the ILM policy being used.

    PUT _ilm/policy/apm-rollover-30-days
    {
      "policy": {
        "phases": {
          "hot": {
            "min_age": "0ms",
            "actions": {
              "rollover": {
                "max_size": "10gb",
                "max_age": "1d"
              },
              "set_priority": {
                "priority": 100
              }
            }
          },
          "warm": {
            "min_age": "5d",
            "actions": {
              "allocate": {
                "number_of_replicas": 1,
                "include": {},
                "exclude": {}
              },
              "readonly": {},
              "set_priority": {
                "priority": 50
              },
              "shrink": {
                "number_of_shards": 1
              }
            }
          }
        }
      }
    }

new index is created everyday and I need to move index older than 5 days to the warm nodes. I can see that the index get rolled over and lifecycle phase have changed to warm.

But the below screenshot says the shards are still in the hot nodes.

Below is the disk usage of the Hot and warm nodes. only 19GB disk usage is there in warm instances and hot instances are still above 75%.

Can someone help me understand if there is anything that I am doing wrong?

Thanks in advance.

[andreidan]

Thank you for using Elasticsearch.

I see you're using elasticsearch 7.10.1 so I want to mention data tiers which are available and might be a way to go forward should you choose to migrate to the specialised data_content, data_hot, data_warm etc node roles and take advantage of the ILM migrate action for automatic data migration between data tiers.

However, with the generic data role you have to implement the data migration between phases using the allocate action and shard allocation awareness.
This blog post describes the shard allocation awareness and allocate action setup.

[Smrithin_N_S]

I updated the ILM policy as

PUT _ilm/policy/apm-rollover-30-days
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_size": "10gb",
            "max_age": "1d"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "warm": {
        "min_age": "5d",
        "actions": {
          "allocate": {
            "number_of_replicas": 1,
            "include": {},
            "exclude": {},
            "require": {
              "data": "warm"
            }
          },
          "readonly": {},
          "set_priority": {
            "priority": 50
          },
          "shrink": {
            "number_of_shards": 1
          }
        }
      }
    }
  }
}

Added the allocate require: data to the ILM policy and during the next rollover warms shards moved to the warm nodes.

Thanks.