ILM action - lifecycle action [migrate] waiting for [1] shards to be moved to the [data_cold] tier

[Kmalik19 (KMalik)]

Hello Everyone,

We have configured ILM policy but it's stucked in warm phase.Here is the details:

ILM Policy definition

{
  "km_ilm" : {
    "version" : 3,
    "modified_date" : "2021-05-31T14:28:26.359Z",
    "policy" : {
      "phases" : {
        "warm" : {
          "min_age" : "60s",
          "actions" : {
            "shrink" : {
              "number_of_shards" : 1
            }
          }
        },
        "cold" : {
          "min_age" : "60s",
          "actions" : {
            "allocate" : {
              "number_of_replicas" : 0,
              "include" : { },
              "exclude" : { },
              "require" : { }
            },
            "searchable_snapshot" : {
              "snapshot_repository" : "km_snapshot",
              "force_merge_index" : true
            },
            "set_priority" : {
              "priority" : 0
            }
          }
        },
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "5mb",
              "max_age" : "60s"
            }
          }
        },
        "delete" : {
          "min_age" : "180s",
          "actions" : {
            "delete" : {
              "delete_searchable_snapshot" : true
            }
          }
        }
      }
    }
  }
}

Here is the index explain :

{
  "indices" : {
    "km_index" : {
      "index" : "km_index",
      "managed" : true,
      "policy" : "km_ilm",
      "lifecycle_date_millis" : 1622468952281,
      "age" : "15.36h",
      "phase" : "warm",
      "phase_time_millis" : 1622469552212,
      "action" : "migrate",
      "action_time_millis" : 1622469552257,
      "step" : "check-migration",
      "step_time_millis" : 1622469552295,
      "step_info" : {
        "message" : "[km_index] lifecycle action [migrate] waiting for [1] shards to be moved to the [data_cold] tier (tier migration preference configuration is [data_cold,data_warm,data_hot])",
        "shards_left_to_allocate" : 1,
        "all_shards_active" : true,
        "number_of_replicas" : 1
      },
      "phase_execution" : {
        "policy" : "km_ilm",
        "phase_definition" : {
          "min_age" : "60s",
          "actions" : {
            "shrink" : {
              "number_of_shards" : 1
            }
          }
        },
        "version" : 3,
        "modified_date_in_millis" : 1622471306359
      }
    }
  }
}

Index Definition

{
  "km_index" : {
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "km_ilm",
          "rollover_alias" : "km_alias1",
          "indexing_complete" : "true"
        },
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_cold,data_warm,data_hot"
            }
          }
        },
        "number_of_shards" : "1",
        "provided_name" : "km_index",
        "creation_date" : "1622466229942",
        "number_of_replicas" : "1",
        "uuid" : "rntE6v_vSduOAAVkfcXcUQ",
        "version" : {
          "created" : "7130099"
        }
      }
    }
  }
}

Could you please assist to move ahead with this.

Thanks,
KM

[andreidan]

Thanks for using Elasticsearch.

I see from the ilm/explain output that one shard was successfully allocated to the cold tier and it's waiting for the replica to be allocated too. Do you have two data_cold nodes in the cluster?

Can you please post the output of GET /_cat/nodes ?