I one index of a datastream showing an ILM error:
java.lang.IllegalStateException: no rollover info found for [.ds-sec-events-2023.08.12-000015] with rollover target [sec-events], the index has not yet rolled over with that target
at org.elasticsearch.xpack.core.ilm.UpdateRolloverLifecycleDateStep.performAction(UpdateRolloverLifecycleDateStep.java:62)
at org.elasticsearch.xpack.ilm.ExecuteStepsUpdateTask.doExecute(ExecuteStepsUpdateTask.java:112)
at org.elasticsearch.xpack.ilm.IndexLifecycleClusterStateUpdateTask.execute(IndexLifecycleClusterStateUpdateTask.java:46)
at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.lambda$static$0(IndexLifecycleRunner.java:64)
at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:840)
at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:407)
at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:243)
at org.elasticsearch.cluster.service.MasterService.access$100(MasterService.java:63)
at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:170)
at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:146)
at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:202)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:262)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:225)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1623)
when I try to use _ilm/explain
it get:
rful011@secesprd01:~$ curl -s --noproxy \* -u elastic -XGET 'https://secesprd02.its.auckland.ac.nz:9200/.ds-sec-events-2023.08.12-000015/_iml/explain'
Enter host password for user 'elastic':
{"_index":".ds-sec-events-2023.08.12-000015","_type":"_iml","_id":"explain","found":false}
which I assume means it can't find the index ?? Is it because it is a backing index for a datastream?
what is weird is that the data stream appears to have rolled over and other indexes have been generated:
something clearly came unstuck -- we had issues with the cluster back then.
the index is still "hot" and 000016 is now cold.