Hello, I have a single-node ES setup that's receiving data from Filebeat running on a separate server.
Since I have Filebeat shipping data from different log sources, I went through the process of manually creating an index template and ILM policy (and then bootstrapping the index) in order to send the data from each source to a separate index.
Over time, after updating Filebeat and Elasticsearch on both the client and ES server side, I've begun seeing a long series of error messages in the Elasticsearch index management view and the elasticsearch log itself. This is an example from the log output:
[2020-11-24T11:51:01,353][ERROR][o.e.x.i.IndexLifecycleRunner] [monitor-elk] policy [lb-web-dataapi] for index [lb-web-dataapi-7.9.0-000002] failed on step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}]. Moving to ERROR step
java.lang.IllegalArgumentException: index.lifecycle.rollover_alias [lb-web-dataapi-7.9.2] does not point to index [lb-web-dataapi-7.9.0-000002]
at org.elasticsearch.xpack.core.ilm.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:114) [x-pack-core-7.9.3.jar:7.9.3]
at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:174) [x-pack-ilm-7.9.3.jar:7.9.3]
at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:329) [x-pack-ilm-7.9.3.jar:7.9.3]
at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggered(IndexLifecycleService.java:267) [x-pack-ilm-7.9.3.jar:7.9.3]
at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:183) [x-pack-core-7.9.3.jar:7.9.3]
at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:211) [x-pack-core-7.9.3.jar:7.9.3]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
Clearly, there is a mismatch between the rollover alias and the index name, that's apparently causing the rollover action to fail. However, the weird part is I don't know where Elasticsearch is getting the old index name "lb-web-dataapi-7.9.0-000002" from, as I've since updated everything to 7.9.3 (on the ES server) and 7.9.2 (on the Filebeat client). In fact, at the same time as these errors are occurring, Elasticsearch is continuing to rollover the newer versions of the index just fine (well it appears to be at least).
Does anyone know how I can find out why ILM is stuck on this old index?