ILM action - lifecycle action [migrate] waiting for [3] shards to be moved to the [data_warm] tier

Elastic Stack Elasticsearch
1

Hello everyone!
I have ILM configured:

"filebeat-prod": {
  "version": 5,
  "modified_date": "2025-01-13T15:41:59.287Z",
  "policy": {
    "phases": {
      "warm": {
        "min_age": "1d",
        "actions": {
          "forcemerge": {
            "max_num_segments": 1
          },
          "shrink": {
            "number_of_shards": 1
          }
        }
      },
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "1d",
            "max_primary_shard_size": "10gb"
          }
        }
      },
      "cold": {
        "min_age": "8d",
        "actions": {}
      }
    }
}

As you can see from the settings, indexes older than 1 day should be moved from hot nodes to warm nodes.
The problem is that some indexes do not move, for example:

{
  "indices": {
    ".ds-logs-prod-legacy-2024.12.09-000083": {
      "index": ".ds-logs-prod-legacy-2024.12.09-000083",
      "managed": true,
      "policy": "filebeat-prod",
      "index_creation_date_millis": 1733720941550,
      "time_since_index_creation": "36.32d",
      "lifecycle_date_millis": 1734607941579,
      "age": "26.05d",
      "phase": "warm",
      "phase_time_millis": 1734694941556,
      "action": "migrate",
      "action_time_millis": 1734694941556,
      "step": "check-migration",
      "step_time_millis": 1734694941956,
      "step_info": {
        "message": "[.ds-logs-prod-legacy-2024.12.09-000083] lifecycle action [migrate] waiting for [3] shards to be moved to the [data_warm] tier (tier migration preference configuration is [data_warm, data_hot])",
        "shards_left_to_allocate": 3,
        "all_shards_active": true,
        "number_of_replicas": 2
      },
      "phase_execution": {
        "policy": "filebeat-prod",
        "phase_definition": {
          "min_age": "1d",
          "actions": {
            "forcemerge": {
              "max_num_segments": 1
            },
            "shrink": {
              "number_of_shards": 1
            }
          }
        },
        "version": 5,
        "modified_date_in_millis": 1736782919287
      }
    }
  }
}

Please tell me how to prevent the problem with stuck indices and prevent hot nodes from overflowing?

2

Hello,
Do you have a Node for data_warm ?

3

Hello,
Thank you for your question.
I have 3 hot nodes and 2 warm nodes. I found out that the indices on hot nodes are stored in 3 shards: 1 primary and 2 replicas. When moving to warm nodes, you need to reduce the number of replicas to one, so that in the end you get two shards. Made the corresponding change to the ILM policy. The problem is solved.