# ILM stuck at check-allocation on transition to Cold Node **URL:** \ **Category:** Elasticsearch\ **Created:** [January 8, 2021, 12:49am UTC](https://discuss.elastic.co/t/ilm-stuck-at-check-allocation-on-transition-to-cold-node/260490 "2021-01-08T00:49:17Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Charles\_Ferguson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/charles_ferguson/32/79551_2.png) [@Charles\_Ferguson](https://discuss.elastic.co/u/Charles_Ferguson)\ **Post date:** [January 8, 2021, 12:49am UTC](https://discuss.elastic.co/t/ilm-stuck-at-check-allocation-on-transition-to-cold-node/260490/1 "2021-01-08T00:49:17Z")
Hi folks, We are having an issue whereour index life cycle policy is not completing the check-allocation phase to the cold nodes. It doesn't seem to be causing any errors just waiting and never completing. Interestingly we don't have the same issue with hot to warm transition just for this transition. I am kind of at a loss on why this is getting stuck and can't find any errors. We are running Elasticsearch 7.9.1 using the ECK operator version 1.3.0. ```auto GET _ilm/policy/companyname-prod-us-global { "companyname-prod-us-global" : { "version" : 11, "modified_date" : "2021-01-06T17:58:03.603Z", "policy" : { "phases" : { "warm" : { "min_age" : "0ms", "actions" : { "allocate" : { "number_of_replicas" : 0, "include" : { }, "exclude" : { }, "require" : { "data" : "warm" } }, "set_priority" : { "priority" : 50 } } }, "cold" : { "min_age" : "30d", "actions" : { "allocate" : { "number_of_replicas" : 0, "include" : { }, "exclude" : { }, "require" : { "data" : "cold" } }, "set_priority" : { "priority" : 20 } } }, "hot" : { "min_age" : "0ms", "actions" : { "rollover" : { "max_size" : "50gb", "max_age" : "1d" }, "set_priority" : { "priority" : 100 } } }, "delete" : { "min_age" : "90d", "actions" : { "delete" : { "delete_searchable_snapshot" : true } } } } } } } ``` ```auto GET nginx-2020.11.27-000002/_ilm/explain { "indices" : { "nginx-2020.11.27-000002" : { "index" : "nginx-2020.11.27-000002", "managed" : true, "policy" : "companyname-prod-us-global", "lifecycle_date_millis" : 1607186458617, "age" : "33.33d", "phase" : "cold", "phase_time_millis" : 1608266450939, "action" : "allocate", "action_time_millis" : 1608266451951, "step" : "check-allocation", "step_time_millis" : 1608266454513, "step_info" : { "message" : "Waiting for [1] shards to be allocated to nodes matching the given filters", "shards_left_to_allocate" : 1, "all_shards_active" : true, "actual_replicas" : 0 }, "phase_execution" : { "policy" : "clio-prod-us-global", "phase_definition" : { "min_age" : "30d", "actions" : { "allocate" : { "number_of_replicas" : 0, "include" : { }, "exclude" : { }, "require" : { "data" : "cold" } }, "set_priority" : { "priority" : 20 } } }, "version" : 11, "modified_date_in_millis" : 1609955883603 } } } } ``` ```auto GET _nodes/*cold*/settings | jq '.nodes | to_entries[] | {name: .value.name, attributes: .value.attributes}' { "name": "prod-es-cold-001-1", "attributes": { "k8s_node_name": "ip-10-110-137-169.ec2.internal", "ml.machine_memory": "6442450944", "ml.max_open_jobs": "20", "xpack.installed": "true", "data": "cold", "transform.node": "true" } } { "name": "prod-es-cold-001-2", "attributes": { "k8s_node_name": "ip-10-110-130-207.ec2.internal", "ml.machine_memory": "6442450944", "ml.max_open_jobs": "20", "xpack.installed": "true", "data": "cold", "transform.node": "true" } } { "name": "prod-es-cold-001-0", "attributes": { "k8s_node_name": "ip-10-110-145-243.ec2.internal", "ml.machine_memory": "6442450944", "ml.max_open_jobs": "20", "xpack.installed": "true", "data": "cold", "transform.node": "true" } } ``` ```auto GET _cat/allocation?v shards disk.indices disk.used disk.avail disk.total disk.percent host ip node 32 359.4gb 361.1gb 508.6gb 869.8gb 41 10.110.138.107 10.110.138.107 prod-es-hot-az2-002-1 51 2.4tb 2.4tb 390.3gb 2.7tb 86 10.110.130.124 10.110.130.124 prod-es-warm-az1-002-0 0 0b 104.3mb 2.7tb 2.7tb 0 10.110.137.136 10.110.137.136 prod-es-cold-001-1 32 526.5gb 527.6gb 342.2gb 869.8gb 60 10.110.139.38 10.110.139.38 prod-es-hot-az2-002-0 31 288.1gb 289.5gb 580.3gb 869.8gb 33 10.110.146.131 10.110.146.131 prod-es-hot-az3-002-1 0 0b 104.3mb 2.7tb 2.7tb 0 10.110.146.225 10.110.146.225 prod-es-cold-001-0 32 755.4gb 755.9gb 113.8gb 869.8gb 86 10.110.145.67 10.110.145.67 prod-es-hot-az3-002-0 76 2.4tb 2.4tb 387.9gb 2.7tb 86 10.110.139.54 10.110.139.54 prod-es-warm-az2-002-0 32 746.3gb 747.7gb 122.1gb 869.8gb 85 10.110.129.120 10.110.129.120 prod-es-hot-az1-002-1 32 416.8gb 418.8gb 450.9gb 869.8gb 48 10.110.128.195 10.110.128.195 prod-es-hot-az1-002-0 0 0b 104.3mb 2.7tb 2.7tb 0 10.110.129.48 10.110.129.48 prod-es-cold-001-2 44 2.4tb 2.4tb 393.1gb 2.7tb 86 10.110.146.45 10.110.146.45 prod-es-warm-az3-002-0 ``` ---
**Author:** ![Charles\_Ferguson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/charles_ferguson/32/79551_2.png) [@Charles\_Ferguson](https://discuss.elastic.co/u/Charles_Ferguson)\ **Post date:** [January 8, 2021, 6:06pm UTC](https://discuss.elastic.co/t/ilm-stuck-at-check-allocation-on-transition-to-cold-node/260490/2 "2021-01-08T18:06:44Z")
Well after all that we finally figured it out. We weren't exposing the zone attributes for the cold storage and the previous stages of the indexes were zone routing aware. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 5, 2021, 6:06pm UTC](https://discuss.elastic.co/t/ilm-stuck-at-check-allocation-on-transition-to-cold-node/260490/3 "2021-02-05T18:06:49Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.