ILM action status - Waiting for all shard copies to be active

Hello,

We have configured ILM policy but it's changing the phase "hot" to "warm" but after that not changing further phases like "cold" and "delete". We see action status "Waiting for all shard copies to be active".

ILM definition

PUT _ilm/policy/acquia-watchdog-ilm-policy
{
"policy": {
"phases": {
"hot": {
"min_age": "0ms",
"actions": {
"set_priority": {
"priority": 0
}
}
},
"warm": {
"min_age": "1d",
"actions": {}
},
"cold": {
"min_age": "7d",
"actions": {}
},
"delete": {
"min_age": "30d",
"actions": {
"delete": {
"delete_searchable_snapshot": true
}
}
}
}
}
}

Could you please assist at this moment.

Thanks and regards
Braham Pal singh

Can you show the output from an explain on the policy?

Hi @warkolm ,
Thanks a ton for your response. Please find the explain output as below:

GET acquia-watchdog-2021.04.01/_ilm/explain
{
"indices" : {
"acquia-watchdog-2021.04.01" : {
"index" : "acquia-watchdog-2021.04.01",
"managed" : true,
"policy" : "acquia-watchdog-ilm-policy",
"lifecycle_date_millis" : 1617235208184,
"age" : "57.2d",
"phase" : "warm",
"phase_time_millis" : 1617321742457,
"action" : "migrate",
"action_time_millis" : 1617321742457,
"step" : "check-migration",
"step_time_millis" : 1617321742925,
"step_info" : {
"message" : "Waiting for all shard copies to be active",
"shards_left_to_allocate" : -1,
"all_shards_active" : false,
"number_of_replicas" : 1
},
"phase_execution" : {
"policy" : "acquia-watchdog-ilm-policy",
"phase_definition" : {
"min_age" : "1d",
"actions" : { }
},
"version" : 2,
"modified_date_in_millis" : 1589258325274
}
}
}
}

Kind regards,
Braham

Is there unallocated shards in the cluster?

Hi @warkolm ,
Not sure, having very limited expertise in ES. Could you please help me how to get unallocated shards in the cluster.

Thanks,
Braham

GET _cat/allocation?v is the best way

Please find the output as below:

GET _cat/allocation?format=json
[
{
"shards" : "276",
"disk.indices" : "47.4gb",
"disk.used" : "82.9gb",
"disk.avail" : "37gb",
"disk.total" : "119.9gb",
"disk.percent" : "69",
"host" : "127.0.0.1",
"ip" : "127.0.0.1",
"node" : "ip-192-168-0-170.eu-west-1.compute.internal"
},
{
"shards" : "188",
"disk.indices" : null,
"disk.used" : null,
"disk.avail" : null,
"disk.total" : null,
"disk.percent" : null,
"host" : null,
"ip" : null,
"node" : "UNASSIGNED"
}
]

Hi @warkolm ,
Is there luck in regard of the shared details in previous comment?

Thanks,
Braham

Hi @warkolm ,
Just to remind about the issue, any suggestion to proceed further.

Thanks,
Braham