# ILM error killing all indexes

**URL:** <https://discuss.elastic.co/t/ilm-error-killing-all-indexes/298993>\
**Category:** Logstash\
**Tags:** ilm-index-lifecycle-management\
**Created:** [March 7, 2022, 4:50pm UTC](https://discuss.elastic.co/t/ilm-error-killing-all-indexes/298993 "2022-03-07T16:50:03Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![cstone1492](https://avatars.discourse-cdn.com/v4/letter/c/9de0a6/32.png) [@cstone1492](https://discuss.elastic.co/u/cstone1492)\
**Post date:** [March 7, 2022, 4:50pm UTC](https://discuss.elastic.co/t/ilm-error-killing-all-indexes/298993/1 "2022-03-07T16:50:03Z")

</div>

The es instance I'm working with crashed, and attempting to restart the kernel leads to the run command immediately being killed (runES.sh: line 1: 92654 Killed ./bin/Elasticsearch).

The log shows this error:

```auto
[2022-03-02T10:40:54,802][INFO][o.e.x.i.IndexLifecycleRunner] [smartmedia] policy [logstash-policy] for index [logstash-testin
dex] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry att
empt [24812]
[2022-03-02T10:50:54,779][ERROR][o.e.x.i.IndexLifecycleRunner] [smartmedia] policy [logstash-policy] for index [logstash-testin
dex] failed on step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}]. Moving to ERROR step
java.lang.IllegalArgumentException: index.lifecycle.rollover_alias [logstash] does not point to index [logstash-testindex]
        at org.elasticsearch.xpack.core.ilm.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:104) [x-pa
ck-core-7.8.1.jar:7.8.1]
        at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:173) [x-pack-ilm-7.8.1.ja
r:7.8.1]
        at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:329) [x-pack-ilm-7.8.1.
jar:7.8.1]
        at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggered(IndexLifecycleService.java:267) [x-pack-ilm-7.8.1.jar:7.
8.1]
        at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:183) [x-pack-core-7.8.1.
jar:7.8.1]
        at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:211) [x-pack-core-7.8
.1.jar:7.8.1]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]

```

If I understand correctly, the problem is this part: [logstash] does not point to index [logstash-testindex], however I do not know how to change an alias for an index if I can't even start the kernel. Is there a config file I need to change instead?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [March 7, 2022, 9:44pm UTC](https://discuss.elastic.co/t/ilm-error-killing-all-indexes/298993/2 "2022-03-07T21:44:40Z")

</div>

Can you share more of your Elasticsearch log please?

---

<div class="post-metadata">

**Author:** ![cstone1492](https://avatars.discourse-cdn.com/v4/letter/c/9de0a6/32.png) [@cstone1492](https://discuss.elastic.co/u/cstone1492)\
**Post date:** [March 7, 2022, 10:47pm UTC](https://discuss.elastic.co/t/ilm-error-killing-all-indexes/298993/3 "2022-03-07T22:47:13Z")

</div>

It's this repeated:

```auto
[2022-03-02T10:40:54,802][INFO][o.e.x.i.IndexLifecycleRunner] [smartmedia] policy [logstash-policy] for index [logstash-testindex] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [24812]
[2022-03-02T10:50:54,779][ERROR][o.e.x.i.IndexLifecycleRunner] [smartmedia] policy [logstash-policy] for index [logstash-testindex] failed on step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}]. Moving to ERROR step
java.lang.IllegalArgumentException: index.lifecycle.rollover_alias [logstash] does not point to index [logstash-testindex]
	at org.elasticsearch.xpack.core.ilm.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:104) [x-pack-core-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:173) [x-pack-ilm-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:329) [x-pack-ilm-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggered(IndexLifecycleService.java:267) [x-pack-ilm-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:183) [x-pack-core-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:211) [x-pack-core-7.8.1.jar:7.8.1]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
	at java.lang.Thread.run(Thread.java:832) [?:?]
[2022-03-02T11:00:54,822][INFO][o.e.x.i.IndexLifecycleRunner] [smartmedia] policy [logstash-policy] for index [logstash-testindex] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [24813]

```

That's the entire Elasticsearch.log file. Is there a different log file that would be helpful?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [March 8, 2022, 12:34am UTC](https://discuss.elastic.co/t/ilm-error-killing-all-indexes/298993/4 "2022-03-08T00:34:51Z")

</div>

That should not cause Elasticsearch to stop/crash.

> [@cstone1492](#):
>
> ```auto
> java.lang.IllegalArgumentException: index.lifecycle.rollover_alias [logstash] does not point to index [logstash-testindex]
> 
> ```

That's the issue.  
So what does your policy look like?

---

<div class="post-metadata">

**Author:** ![cstone1492](https://avatars.discourse-cdn.com/v4/letter/c/9de0a6/32.png) [@cstone1492](https://discuss.elastic.co/u/cstone1492)\
**Post date:** [March 8, 2022, 2:58pm UTC](https://discuss.elastic.co/t/ilm-error-killing-all-indexes/298993/5 "2022-03-08T14:58:07Z")

</div>

I don't know as I'm not the person who set up or used logstash. How would I check the policy of an index when Elasticsearch isn't running? the properties file for logstash doesn't mention any index alias (see below)

```auto
status = error

appender.console.type = Console
appender.console.name = console
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n

######## Server JSON ############################
appender.rolling.type = RollingFile
appender.rolling.name = rolling
appender.rolling.fileName = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}_server.json
appender.rolling.layout.type = ESJsonLayout
appender.rolling.layout.type_name = server

appender.rolling.filePattern = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}-%d{yyyy-MM-dd}-%i.json.gz
appender.rolling.policies.type = Policies
appender.rolling.policies.time.type = TimeBasedTriggeringPolicy
appender.rolling.policies.time.interval = 1
appender.rolling.policies.time.modulate = true
appender.rolling.policies.size.type = SizeBasedTriggeringPolicy
appender.rolling.policies.size.size = 128MB
appender.rolling.strategy.type = DefaultRolloverStrategy
appender.rolling.strategy.fileIndex = nomax
appender.rolling.strategy.action.type = Delete
appender.rolling.strategy.action.basepath = ${sys:es.logs.base_path}
appender.rolling.strategy.action.condition.type = IfFileName
appender.rolling.strategy.action.condition.glob = ${sys:es.logs.cluster_name}-*
appender.rolling.strategy.action.condition.nested_condition.type = IfAccumulatedFileSize
appender.rolling.strategy.action.condition.nested_condition.exceeds = 2GB
################################################
######## Server - old style pattern ###########
appender.rolling_old.type = RollingFile
appender.rolling_old.name = rolling_old
appender.rolling_old.fileName = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}.log
appender.rolling_old.layout.type = PatternLayout
appender.rolling_old.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n

appender.rolling_old.filePattern = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}-%d{yyyy-MM-dd}-%i.log.gz
appender.rolling_old.policies.type = Policies
appender.rolling_old.policies.time.type = TimeBasedTriggeringPolicy
appender.rolling_old.policies.time.interval = 1
appender.rolling_old.policies.time.modulate = true
appender.rolling_old.policies.size.type = SizeBasedTriggeringPolicy
appender.rolling_old.policies.size.size = 128MB
appender.rolling_old.strategy.type = DefaultRolloverStrategy
appender.rolling_old.strategy.fileIndex = nomax
appender.rolling_old.strategy.action.type = Delete
appender.rolling_old.strategy.action.basepath = ${sys:es.logs.base_path}
appender.rolling_old.strategy.action.condition.type = IfFileName

```

---

<div class="post-metadata">

**Author:** ![cstone1492](https://avatars.discourse-cdn.com/v4/letter/c/9de0a6/32.png) [@cstone1492](https://discuss.elastic.co/u/cstone1492)\
**Post date:** [March 8, 2022, 5:16pm UTC](https://discuss.elastic.co/t/ilm-error-killing-all-indexes/298993/6 "2022-03-08T17:16:10Z")

</div>

I'd like to add that trying to run Elasticsearch currently leads to the process immediately being killed:

```auto
esStack/elasticsearch-7.8.1$ bash runES.sh
[0.002s][warning][logging] Output options for existing outputs are ignored.
runES.sh: line 1: 102120 Killed ./bin/elasticsearch

```

with nothing being added to the logs

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [March 8, 2022, 10:09pm UTC](https://discuss.elastic.co/t/ilm-error-killing-all-indexes/298993/7 "2022-03-08T22:09:27Z")

</div>

These two issues are not related.

> [@cstone1492](#):
>
> `bash runES.sh`

What is this script? It's not something that we provide.

---

<div class="post-metadata">

**Author:** ![cstone1492](https://avatars.discourse-cdn.com/v4/letter/c/9de0a6/32.png) [@cstone1492](https://discuss.elastic.co/u/cstone1492)\
**Post date:** [March 9, 2022, 2:56pm UTC](https://discuss.elastic.co/t/ilm-error-killing-all-indexes/298993/8 "2022-03-09T14:56:33Z")

</div>

It contains the following:

```auto
./bin/elasticsearch

```

Running directly with ./bin/Elasticsearch causes the same warning with the process being immediately killed

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [March 9, 2022, 9:39pm UTC](https://discuss.elastic.co/t/ilm-error-killing-all-indexes/298993/9 "2022-03-09T21:39:20Z")

</div>

What user are you running it at?  
What happens when you run `./bin/elasticsearch` directly?

---

<div class="post-metadata">

**Author:** ![cstone1492](https://avatars.discourse-cdn.com/v4/letter/c/9de0a6/32.png) [@cstone1492](https://discuss.elastic.co/u/cstone1492)\
**Post date:** [March 10, 2022, 3:06pm UTC](https://discuss.elastic.co/t/ilm-error-killing-all-indexes/298993/10 "2022-03-10T15:06:59Z")

</div>

I don't understand the first question, however as I said above when I run ./bin/Elasticsearch I get the same error, with the process being immediately killed.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 7, 2022, 3:07pm UTC](https://discuss.elastic.co/t/ilm-error-killing-all-indexes/298993/11 "2022-04-07T15:07:25Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.