ILM keeps losing track of indices

[cawoodm (Marc)]

We keep seeing the issue described here where Elasticsearch is not rolling over indices properly.

[2021-09-13T12:39:16,671][ERROR][o.e.x.i.IndexLifecycleRunner] [ecom-repository01] policy [test-policy] for index [errors-test-000021] failed on step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}]. Moving to ERROR step
java.lang.IllegalArgumentException: index.lifecycle.rollover_alias [errors-test] does not point to index [errors-test-000021]
at org.elasticsearch.xpack.core.ilm.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:104) [x-pack-core-7.7.0.jar:7.7.0]
at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:173) [x-pack-ilm-7.7.0.jar:7.7.0]
at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:329) [x-pack-ilm-7.7.0.jar:7.7.0]
at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggered(IndexLifecycleService.java:267) [x-pack-ilm-7.7.0.jar:7.7.0]
at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:183) [x-pack-core-7.7.0.jar:7.7.0]

1. How can we resolve this permanently? My feeling is ILM in general is a bit flaky.
2. Is there a way to query such orphaned indices so we can re-attach them?

[sholzhauer]

Hi cawoodm,

This looks to me like an issue in your index templates.

Your ILM is looking for the "alias" errors-test". Meaning your index has an ILM assigned which is looking for an alias which it doesn't have.

Have a look at your index template(s) to see if the alias is assigned upon creation of the index.