ILM says it is waiting to allocate 1 shard (but it isn't)

[DanRoscigno]

Hi Folks,
I see this for all of my warm indices:

Here is the phase definition:
{
"policy": "filebeat-7.5.2",
"phase_definition": {
"min_age": "0ms",
"actions": {
"allocate": {
"include": {},
"exclude": {},
"require": {
"box_type": "warm"
}
}
}
},
"version": 2,
"modified_date_in_millis": 1580914699313
}

and here is the policy:
PUT _ilm/policy/filebeat-7.5.2
{
"policy": {
"phases": {
"hot": {
"min_age": "0ms",
"actions": {
"rollover": {
"max_age": "3h",
"max_size": "50gb"
}
}
},
"warm": {
"actions": {
"allocate": {
"include": {},
"exclude": {},
"require": {
"box_type": "warm"
}
}
}
},
"delete": {
"min_age": "2d",
"actions": {
"delete": {}
}
}
}
}
}

Here is the output of GET /_cluster/allocation/explain
{
"error": {
"root_cause": [
{
"type": "illegal_argument_exception",
"reason": "unable to find any unassigned shards to explain [ClusterAllocationExplainRequest[useAnyUnassignedShard=true,includeYesDecisions?=false]"
}
],
"type": "illegal_argument_exception",
"reason": "unable to find any unassigned shards to explain [ClusterAllocationExplainRequest[useAnyUnassignedShard=true,includeYesDecisions?=false]"
},
"status": 400
}

Stack monitoring shows zero unassigned shards, that my warm node has 67% free space, and my hot nodes have 45% and 64% free space.

I am running 7.5.2 Filebeat, Logstash, Elasticsearch, and Kibana on Centos 7.0

[DanRoscigno]

Oops.

I bet it is because I have only one warm node. I will add another warm and see if the message goes away. I think the replicas for all of my warm indices are on my hot boxes at the moment.

[DanRoscigno]

That was it, I added a second warm node and everything is groovy. I will leave this here in case I forget again

[Doc_Kaos (Ken R)]

Should mark that as "Solution"

[DanRoscigno]

Thanks, I was trying to figure out how to do that!