Index lifecycle management runs into IndexNotFoundException

Elastic Stack Elasticsearch
1

Hi, I'm starting to use index lifecycle management policy to delete index that's been created for xx time. Here's the policy:

{ "policy": {
    "phases": {
        "hot": {
            "actions": {}},
        "delete": {
            "min_age": xx,
            "actions": {
                "delete": {}}}}}

The deletion min_age is a variable, and the policy is in an index template.
In test, I set the delete min_age to be 50ms, create an index template that has the policy, create an index that follows the template, and then wait for 50ms and check in a loop whether the index is deleted.
But I sometimes ran into

org.elasticsearch.index.IndexNotFoundException: no such index

other times the test passes.
Here's the more detailed log:

[2019-02-22T15:39:56,990][INFO ][o.e.c.m.MetaDataIndexTemplateService] [ejsJGon] adding template [template_test-1550849985826781595] for index patterns [test-1550849985826781595-*]
[2019-02-22T15:39:57,656][INFO ][o.e.c.m.MetaDataCreateIndexService] [ejsJGon] [test-1550849985826781595-2019-02-11] creating index, cause [api], templates [template_test-1550849985826781595], shards [5]/[2], mappings [_doc]
[2019-02-22T15:40:05,073][INFO ][o.e.c.m.MetaDataDeleteIndexService] [ejsJGon] [test-1550849985826781595-2019-02-11/yD0Ns76GR3WddMX_sjRoNQ] deleting index
[2019-02-22T15:40:05,207][ERROR][o.e.x.i.IndexLifecycleRunner] [ejsJGon] policy [test-1550849985826781595_policy] for index [test-1550849985826781595-2019-02-11] failed on step [{"phase":"delete","action":"delete","name":"delete"}]. Moving to ERROR step
org.elasticsearch.index.IndexNotFoundException: no such index
	at org.elasticsearch.cluster.metadata.MetaData.getIndexSafe(MetaData.java:614) ~[elasticsearch-6.6.0.jar:6.6.0]
	at org.elasticsearch.cluster.metadata.MetaDataDeleteIndexService.lambda$deleteIndices$0(MetaDataDeleteIndexService.java:93) ~[elasticsearch-6.6.0.jar:6.6.0]
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) ~[?:?]
	at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1603) ~[?:?]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) ~[?:?]
	at org.elasticsearch.cluster.metadata.MetaDataDeleteIndexService.deleteIndices(MetaDataDeleteIndexService.java:93) ~[elasticsearch-6.6.0.jar:6.6.0]
	at org.elasticsearch.cluster.metadata.MetaDataDeleteIndexService$1.execute(MetaDataDeleteIndexService.java:83) ~[elasticsearch-6.6.0.jar:6.6.0]
	at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47) ~[elasticsearch-6.6.0.jar:6.6.0]
	at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:643) ~[elasticsearch-6.6.0.jar:6.6.0]
	at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:272) ~[elasticsearch-6.6.0.jar:6.6.0]
	at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:202) [elasticsearch-6.6.0.jar:6.6.0]
	at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:137) [elasticsearch-6.6.0.jar:6.6.0]
	at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-6.6.0.jar:6.6.0]
	at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-6.6.0.jar:6.6.0]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) [elasticsearch-6.6.0.jar:6.6.0]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [elasticsearch-6.6.0.jar:6.6.0]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) [elasticsearch-6.6.0.jar:6.6.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]

So in the log, the IndexLifecycleRunner can specify the correct index name, but it cannot find the index.
Is that because 50ms is too small? Can I get some help regarding this? Thanks!

2

I do not see the point in setting min_age to 50ms. What is the usecase? What are you trying to test or achieve?

3

Thanks for replying. I'm trying to test if my ilm policy definition is correct and how it's integrated into index template is correct in my code. By setting the min_age to a small value, I can test whether it's working in a go unit test. The use case is eventually to use ilm to delete index older than 28 days. Right now we're using curator to delete old indices, and would like to replace it with the ilm.

4

I do not that think that is going to work as I believe conditions are checked periodically at an interval far greater than the timeout you have specified. Indices are expected to be reasonably long lived, so it does not make any sense checking condition extremely frequently if this adds load to the cluster.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.