Ilm rollover error on datastream index

[Russell_Fulton]

I one index of a datastream showing an ILM error:

java.lang.IllegalStateException: no rollover info found for [.ds-sec-events-2023.08.12-000015] with rollover target [sec-events], the index has not yet rolled over with that target
	at org.elasticsearch.xpack.core.ilm.UpdateRolloverLifecycleDateStep.performAction(UpdateRolloverLifecycleDateStep.java:62)
	at org.elasticsearch.xpack.ilm.ExecuteStepsUpdateTask.doExecute(ExecuteStepsUpdateTask.java:112)
	at org.elasticsearch.xpack.ilm.IndexLifecycleClusterStateUpdateTask.execute(IndexLifecycleClusterStateUpdateTask.java:46)
	at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.lambda$static$0(IndexLifecycleRunner.java:64)
	at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:840)
	at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:407)
	at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:243)
	at org.elasticsearch.cluster.service.MasterService.access$100(MasterService.java:63)
	at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:170)
	at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:146)
	at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:202)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718)
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:262)
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:225)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1623)

when I try to use _ilm/explain it get:

rful011@secesprd01:~$ curl -s --noproxy \* -u elastic  -XGET 'https://secesprd02.its.auckland.ac.nz:9200/.ds-sec-events-2023.08.12-000015/_iml/explain' 
Enter host password for user 'elastic':
{"_index":".ds-sec-events-2023.08.12-000015","_type":"_iml","_id":"explain","found":false}

which I assume means it can't find the index ?? Is it because it is a backing index for a datastream?

what is weird is that the data stream appears to have rolled over and other indexes have been generated:

something clearly came unstuck -- we had issues with the cluster back then.

the index is still "hot" and 000016 is now cold.

[stephenb (Stephen Brown)]

@Russell_Fulton

_iml

@Russell_Fulton

https://secesprd02.its.auckland.ac.nz:9200/.ds-sec-events-2023.08.12-000015/_iml/explain

Typo

_ilm

[Russell_Fulton]

Thanks! sigh... I am hopeless at spotting typos!

OK the _ilm/explain returned the error message above along with the general information about the index.

I think the fundamental problem here is I lost one shard of this index when it was the head of the data datastream . To recover it I had to delete it forcing the datasteam to start a new head, then restored the the index from backup and linked it back into the datastream (with help from this forum -- probably @stephenb : ).

I suspect this left the index as an orphan as far as the ilm process is concerned.

Any thoughts on how to correct this? Does it matter?