# ILM turn unstable the cluster. Heap Overflow

**URL:** <https://discuss.elastic.co/t/ilm-turn-unstable-the-cluster-heap-overflow/205242>\
**Category:** Elasticsearch\
**Created:** [October 25, 2019, 11:15am UTC](https://discuss.elastic.co/t/ilm-turn-unstable-the-cluster-heap-overflow/205242 "2019-10-25T11:15:02Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Emiliano\_Baum](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/emiliano_baum/32/5186_2.png) [@Emiliano\_Baum](https://discuss.elastic.co/u/Emiliano_Baum)\
**Post date:** [October 25, 2019, 11:15am UTC](https://discuss.elastic.co/t/ilm-turn-unstable-the-cluster-heap-overflow/205242/1 "2019-10-25T11:15:03Z")

</div>

when i try to execute:

```
curl -X POST "xx.xx.xxx.xx:9200/_ilm/stop?pretty

```

the result is:  
"type" : "process\_cluster\_event\_timeout\_exception"

I think the ILM operation schedule generate an overflow heap error:

[2019-10-25T00:53:10,313][WARN][o.e.b.ElasticsearchUncaughtExceptionHandler] [CGSS-CLUSTER01-MASTER-1] uncaught exception in thread [elasticsearch[CGSS-CL  
USTER01-MASTER-1][management][T#1]]  
java.lang.IllegalStateException: expected index [quorum01-20180202] with policy [time\_close] to have current step consistent with provided step key (null)  
but it was {"phase":"new","action":"init","name":"init"}  
at org.elasticsearch.xpack.indexlifecycle.IndexLifecycleRunner.maybeRunAsyncAction(IndexLifecycleRunner.java:167) ~[?:?]  
at org.elasticsearch.xpack.indexlifecycle.IndexLifecycleService.onMaster(IndexLifecycleService.java:123) ~[?:?]  
at org.elasticsearch.cluster.service.ClusterApplierService$OnMasterRunnable.run(ClusterApplierService.java:616) ~[elasticsearch-6.6.0.jar:6.6.0]  
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) ~[elasticsearch-6.6.0.jar:6.6.0]  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0\_31]  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0\_31]  
at java.lang.Thread.run(Thread.java:745) [?:1.8.0\_31]  
[2019-10-25T00:53:10,754][DEBUG][o.e.a.a.i.c.TransportCreateIndexAction] [CGSS-CLUSTER01-MASTER-1] timed out while retrying [indices:admin/create] after fa  
ilure (timeout [30s])  
[2019-10-25T00:53:10,755][DEBUG][o.e.a.a.i.c.TransportCreateIndexAction] [CGSS-CLUSTER01-MASTER-1] timed out while retrying [indices:admin/create] after fa  
ilure (timeout [30s])

Any recommendation? Before i configure ILM, i use to run my own program in python to close old indexes using a time mask and i never had any problems with the cluster.

---

<div class="post-metadata">

**Author:** ![Emiliano\_Baum](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/emiliano_baum/32/5186_2.png) [@Emiliano\_Baum](https://discuss.elastic.co/u/Emiliano_Baum)\
**Post date:** [October 29, 2019, 2:28pm UTC](https://discuss.elastic.co/t/ilm-turn-unstable-the-cluster-heap-overflow/205242/2 "2019-10-29T14:28:21Z")

</div>

Finally i modified the index settings using curl:

```
curl -X PUT " **.**. ***.*** :9200/ **** -20191025/_settings?pretty" -H 'Content-Type: application/json' -d'{"index":{"lifecycle":{"name":""}}}'

```

Then i enable allocation for primary and then for all...  
The index life cycle managment, used only configuration for cold phase, closing index older than 30 days generates a heap overflow. I saw on the master nodes the use of heap memory is not balanced, and the ilm process generate the overflow and kill master nodes until reach the limit established at the .yml config file.

Probably is not a very "clean" solution, but works... the cluster is now full operative.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 26, 2019, 2:28pm UTC](https://discuss.elastic.co/t/ilm-turn-unstable-the-cluster-heap-overflow/205242/3 "2019-11-26T14:28:22Z")

</div>

Closed on November 26, 2019, 2:28pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.