# ILM Waiting for Allocation on Warm Node

**URL:** <https://discuss.elastic.co/t/ilm-waiting-for-allocation-on-warm-node/208285>\
**Category:** Elasticsearch\
**Tags:** ilm-index-lifecycle-management\
**Created:** [November 18, 2019, 10:16am UTC](https://discuss.elastic.co/t/ilm-waiting-for-allocation-on-warm-node/208285 "2019-11-18T10:16:47Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![anilozyurt](https://avatars.discourse-cdn.com/v4/letter/a/ac8455/32.png) [@anilozyurt](https://discuss.elastic.co/u/anilozyurt)\
**Post date:** [November 18, 2019, 10:16am UTC](https://discuss.elastic.co/t/ilm-waiting-for-allocation-on-warm-node/208285/1 "2019-11-18T10:16:48Z")

</div>

Hi,

I'm using 3 nodes cluster and monthly indices with 6 primary and 1 replica shards.

es-node-1 Hot node  
es-node-2 Warm node  
es-node-3 Cold node

Using the ILM policy with no rollover and the monthly indices stuck at allocation action onto Warm node. The indices waiting about 4 days to allocate.

I tested this lifecycle way a month ago and it worked fine. But now the lifecycle is not working as expected. How can I resolve this issue?

ILM explain:

> Blockquote  
> {  
> "indices" : {  
> "document-2019.10" : {  
> "index" : "document-2019.10",  
> "managed" : true,  
> "policy" : "grid-ilm-policy",  
> "lifecycle\_date\_millis" : 1571048211487,  
> "phase" : "warm",  
> "phase\_time\_millis" : 1573813256180,  
> "action" : "allocate",  
> "action\_time\_millis" : 1573813856327,  
> "step" : "check-allocation",  
> "step\_time\_millis" : 1573813856438,  
> "step\_info" : {  
> "message" : "Waiting for [6] shards to be allocated to nodes matching the given filters",  
> "shards\_left\_to\_allocate" : 6,  
> "all\_shards\_active" : true,  
> "actual\_replicas" : 1  
> },  
> "phase\_execution" : {  
> "policy" : "grid-ilm-policy",  
> "phase\_definition" : {  
> "min\_age" : "32d",  
> "actions" : {  
> "allocate" : {  
> "include" : { },  
> "exclude" : { },  
> "require" : {  
> "box\_type" : "warm"  
> }  
> },  
> "set\_priority" : {  
> "priority" : 50  
> },  
> "shrink" : {  
> "number\_of\_shards" : 1  
> }  
> }  
> },  
> "version" : 9,  
> "modified\_date\_in\_millis" : 1571038556385  
> }  
> }  
> }  
> }

Index settings:

> Blockquote  
> {  
> "document-2019.10" : {  
> "settings" : {  
> "index" : {  
> "lifecycle" : {  
> "name" : "grid-ilm-policy"  
> },  
> "routing" : {  
> "allocation" : {  
> "require" : {  
> "box\_type" : "warm"  
> }  
> }  
> },  
> "number\_of\_shards" : "6",  
> "provided\_name" : "document-2019.10",  
> "creation\_date" : "1571048211487",  
> "priority" : "50",  
> "number\_of\_replicas" : "1",  
> "uuid" : "9YQB0XJ5SU2aT\_L1ZLIWuw",  
> "version" : {  
> "created" : "7030299"  
> }  
> }  
> }  
> }  
> }

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 19, 2019, 6:48am UTC](https://discuss.elastic.co/t/ilm-waiting-for-allocation-on-warm-node/208285/2 "2019-11-19T06:48:43Z")

</div>

Use the [cluster allocation explain API](https://www.elastic.co/guide/en/elasticsearch/reference/7.4/cluster-allocation-explain.html) to find out why the indices are not relocating. Do you have enough disk space on the nodes that are to receive the shards? By default no shards can be allocated to a node once it gets more than 85% full.

---

<div class="post-metadata">

**Author:** ![anilozyurt](https://avatars.discourse-cdn.com/v4/letter/a/ac8455/32.png) [@anilozyurt](https://discuss.elastic.co/u/anilozyurt)\
**Post date:** [November 19, 2019, 7:27am UTC](https://discuss.elastic.co/t/ilm-waiting-for-allocation-on-warm-node/208285/3 "2019-11-19T07:27:03Z")

</div>

Hi Christian,

Thanks for your reply. I used this API but it turns that there is no unnassigned shards to explain.

GET /\_cluster/allocation/explain

> {  
> "error": {  
> "root\_cause": [  
> {  
> "type": "remote\_transport\_exception",  
> "reason": "[es-node-3][10.200.0.6:9300][cluster:monitor/allocation/explain]"  
> }  
> ],  
> "type": "illegal\_argument\_exception",  
> "reason": "unable to find any unassigned shards to explain [ClusterAllocationExplainRequest[useAnyUnassignedShard=true,includeYesDecisions?=false]"  
> },  
> "status": 400  
> }

GET /\_cat/allocation?v

> shards disk.indices disk.used disk.avail disk.total disk.percent host ip node  
> 202 59.1gb 80.1gb 16.5gb 96.6gb 82 10.200.0.4 10.200.0.4 es-node-1  
> 202 61gb 80.4gb 114.5gb 195gb 41 10.200.0.6 10.200.0.6 es-node-3  
> 204 56.1gb 75.1gb 70.7gb 145.8gb 51 10.200.0.5 10.200.0.5 es-node-2

The disk percents on the nodes:  
es-node-1 \> %82  
es-node-2 \> %51  
es-node-3 \> %41

So the disk watermark(%85) is not issued on any of the nodes. I tested this ILM before with this configuration and it was worked fine but I'm waiting for days now.

---

<div class="post-metadata">

**Author:** ![ericv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ericv/32/54588_2.png) [@ericv](https://discuss.elastic.co/u/ericv)\
**Post date:** [December 2, 2019, 10:55pm UTC](https://discuss.elastic.co/t/ilm-waiting-for-allocation-on-warm-node/208285/4 "2019-12-02T22:55:16Z")

</div>

Hello Anil,

Have you found a solution for your problem? I'm facing exactly the same issue on my ES Cloud Deployment. Indices are not rolling over, stating they're waiting for a node to become available.

Kind regards,

Eric V.

---

<div class="post-metadata">

**Author:** ![anilozyurt](https://avatars.discourse-cdn.com/v4/letter/a/ac8455/32.png) [@anilozyurt](https://discuss.elastic.co/u/anilozyurt)\
**Post date:** [December 3, 2019, 6:29am UTC](https://discuss.elastic.co/t/ilm-waiting-for-allocation-on-warm-node/208285/5 "2019-12-03T06:29:12Z")

</div>

Hi ericv,

I'm using the policy to shrink indices to 1primary&1replica shard. So the primary shard assigning on the warm node but we need a second warm node to replica shard to assign. Elasticsearch does not permit the primary and replica shards to assign on same node.

So you have to add minimum 2 warm nodes to 1p&1r shard or have to store the shards with no replica with 1 primary shard. I solved my issue with adding a "warm" node to my cluster. Referring this you have to add one "cold" node too if you want to store the indices with 1 replica.

---

<div class="post-metadata">

**Author:** ![ericv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ericv/32/54588_2.png) [@ericv](https://discuss.elastic.co/u/ericv)\
**Post date:** [December 3, 2019, 8:16am UTC](https://discuss.elastic.co/t/ilm-waiting-for-allocation-on-warm-node/208285/6 "2019-12-03T08:16:08Z")

</div>

Thanks Anil for your response.

I already have 2 hot and 2 warm nodes so that wasn't the issue, and there was also more than enough disk space available.  
My issue luckily got solved when I upgraded my deployment to 7.5.0 this morning.

Kind regards,

Eric V.

---

<div class="post-metadata">

**Author:** ![Kevin\_Tibi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kevin_tibi/32/54431_2.png) [@Kevin\_Tibi](https://discuss.elastic.co/u/Kevin_Tibi)\
**Post date:** [December 5, 2019, 10:13am UTC](https://discuss.elastic.co/t/ilm-waiting-for-allocation-on-warm-node/208285/7 "2019-12-05T10:13:16Z")

</div>

Same issue here, solved by upgrade to 7.5.0

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 2, 2020, 10:13am UTC](https://discuss.elastic.co/t/ilm-waiting-for-allocation-on-warm-node/208285/8 "2020-01-02T10:13:18Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.