# Unable to allocate shards in ILM cold phase

**URL:** <https://discuss.elastic.co/t/unable-to-allocate-shards-in-ilm-cold-phase/273789>\
**Category:** Elasticsearch\
**Tags:** ilm-index-lifecycle-management\
**Created:** [May 24, 2021, 7:24am UTC](https://discuss.elastic.co/t/unable-to-allocate-shards-in-ilm-cold-phase/273789 "2021-05-24T07:24:52Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![askids](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/askids/32/22252_2.png) [@askids](https://discuss.elastic.co/u/askids)\
**Post date:** [May 24, 2021, 7:24am UTC](https://discuss.elastic.co/t/unable-to-allocate-shards-in-ilm-cold-phase/273789/1 "2021-05-24T07:24:52Z")

</div>

hi,  
I have a 2 node cluster running 7.10.2. I have defined ILM policy to move index from hot-warm-cold phases. However the transition from warm to cold phase is not happening. I have not explicitly set any node roles. So I expect all nodes to perform all roles. But when I run explain on cluster allocation, I see below error that "index.routing.allocation.require" - cold condition was not met. How do I fix this issue?

```
"node_allocation_decisions" : [
    {
      "node_id" : "GNR2p8B7QaeL86xWa6o3kQ",
      "node_name" : "mynodename",
      "transport_address" : "123.123.123.123:9300",
      "node_attributes" : {
        "xpack.installed" : "true",
        "transform.node" : "true"
      },
      "node_decision" : "no",
      "deciders" : [
        {
          "decider" : "filter",
          "decision" : "NO",
          "explanation" : """node does not match index setting [index.routing.allocation.require] filters [type:"cold"]"""
        }
      ]

```

Thanks!

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [May 24, 2021, 7:52am UTC](https://discuss.elastic.co/t/unable-to-allocate-shards-in-ilm-cold-phase/273789/2 "2021-05-24T07:52:00Z")

</div>

What does your policy look like, and what's the output from an explain of it?  
What's the output from `GET /_cat/nodeattrs?v`?

---

<div class="post-metadata">

**Author:** ![askids](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/askids/32/22252_2.png) [@askids](https://discuss.elastic.co/u/askids)\
**Post date:** [May 24, 2021, 8:14am UTC](https://discuss.elastic.co/t/unable-to-allocate-shards-in-ilm-cold-phase/273789/3 "2021-05-24T08:14:33Z")

</div>

nodeattrs shows this output.

```
node host ip attr value
mynodename1 myhost1.mydomain.com 123.123.123.123 xpack.installed true
mynodename1 myhost1.mydomain.com 123.123.123.123 transform.node true
mynodename2 myhost2.mydomain.com 123.123.123.124 xpack.installed true
mynodename2 myhost2.mydomain.com 123.123.123.124 transform.node true
```

---

<div class="post-metadata">

**Author:** ![askids](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/askids/32/22252_2.png) [@askids](https://discuss.elastic.co/u/askids)\
**Post date:** [May 24, 2021, 8:19am UTC](https://discuss.elastic.co/t/unable-to-allocate-shards-in-ilm-cold-phase/273789/4 "2021-05-24T08:19:53Z")

</div>

My ILM policy

```
PUT _ilm/policy/my_data_rollover_policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_size": "3gb",
            "max_age": "15d"
          }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "allocate": {
            "number_of_replicas": 2,
            "include": {},
            "exclude": {}
          },
          "shrink": {
            "number_of_shards": 5
          },
          "forcemerge": {
            "max_num_segments": 1
          }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {
          "allocate": {
            "include": {},
            "exclude": {},
            "require": {
              "type": "cold"
            }
          }
        }
      },
      "delete": {
        "min_age": "60d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}
```

---

<div class="post-metadata">

**Author:** ![askids](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/askids/32/22252_2.png) [@askids](https://discuss.elastic.co/u/askids)\
**Post date:** [May 25, 2021, 2:22pm UTC](https://discuss.elastic.co/t/unable-to-allocate-shards-in-ilm-cold-phase/273789/5 "2021-05-25T14:22:33Z")

</div>

hi @warkolm,

Any suggestions?

Thanks!

---

<div class="post-metadata">

**Author:** ![askids](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/askids/32/22252_2.png) [@askids](https://discuss.elastic.co/u/askids)\
**Post date:** [June 9, 2021, 11:54am UTC](https://discuss.elastic.co/t/unable-to-allocate-shards-in-ilm-cold-phase/273789/6 "2021-06-09T11:54:12Z")

</div>

hi @warkolm,

Any suggestions? We are stuck on this issue.

Thanks!

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [June 9, 2021, 9:11pm UTC](https://discuss.elastic.co/t/unable-to-allocate-shards-in-ilm-cold-phase/273789/7 "2021-06-09T21:11:38Z")

</div>

You'll need to check your config to make sure that you have nodes with the cold role allocated.

---

<div class="post-metadata">

**Author:** ![askids](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/askids/32/22252_2.png) [@askids](https://discuss.elastic.co/u/askids)\
**Post date:** [June 12, 2021, 5:00pm UTC](https://discuss.elastic.co/t/unable-to-allocate-shards-in-ilm-cold-phase/273789/8 "2021-06-12T17:00:56Z")

</div>

Like i had mentioned earlier, i have not put any specific node roles. When I run \_nodes, it shows that all nodes have data\_cold role. Do I need to rerun the policy after upgrading ES so that policy will pick up new role names? Currently we have upgraded to 7.10.2, but the policy was created on 7.2.0 and then updated on 7.8.1. So you think that could be the issue?

```auto
"roles" : [
        "data",
        "data_cold",
        "data_content",
        "data_hot",
        "data_warm",
        "ingest",
        "master",
        "remote_cluster_client",
        "transform"
      ],

```

---

<div class="post-metadata">

**Author:** ![andreidan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andreidan/32/78167_2.png) [@andreidan](https://discuss.elastic.co/u/andreidan)\
**Post date:** [June 14, 2021, 8:36am UTC](https://discuss.elastic.co/t/unable-to-allocate-shards-in-ilm-cold-phase/273789/9 "2021-06-14T08:36:06Z")

</div>

The `allocate` action in the cold phase is relying on node _attributes_.

```auto
        "actions": {
          "allocate": {
            "include": {},
            "exclude": {},
            "require": {
              "type": "cold"
            }
          }

```

This configuration is looking for a node attribute with name `type` and value `cold`. I believe you need to tag your nodes with the attribute `type` and the corresponding value (`warm` for warm nodes, `cold` for cold nodes etc) and update your policy to also use node attributes in the warm phase **as well** (as it's currently not defining any allocation rules in the `allocate` action so ILM will [automatically inject a `migrate` action](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-migrate.html#ilm-migrate)).  
Also be aware that new indices will receive the `_tier_preference` routing automatically as [described here](https://www.elastic.co/guide/en/elasticsearch/reference/current/data-tiers.html#data-tier-allocation) - there're ways to opt out of this as well, described in the same post (you usually don't want to mix data tier allocation and custom node attribute values)

Alternatively you can migrate to only using node roles allocation (data tiers) as described in the [migration guide](https://www.elastic.co/guide/en/elasticsearch/reference/current/migrate-index-allocation-filters.html).

---

<div class="post-metadata">

**Author:** ![askids](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/askids/32/22252_2.png) [@askids](https://discuss.elastic.co/u/askids)\
**Post date:** [June 18, 2021, 6:26pm UTC](https://discuss.elastic.co/t/unable-to-allocate-shards-in-ilm-cold-phase/273789/10 "2021-06-18T18:26:49Z")

</div>

Thanks @andreidan I updated the ILM policy to remove type requirement. After that, the rollover worked properly.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 16, 2021, 6:27pm UTC](https://discuss.elastic.co/t/unable-to-allocate-shards-in-ilm-cold-phase/273789/11 "2021-07-16T18:27:00Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.