ILM action failed "check-rollover-ready,Moving to ERROR step"?

Elastic Stack Elasticsearch
1

I found that my es cluster is always reporting an error, he said that the ILM work failed, according to the prompt "check-rollover-ready", I didn't understand what it needs to identify, and the target index is the system index, I haven't made any changes, why is it Failed?

policy [ilm-history-ilm-policy] for index [.ds-ilm-history-5-2021.10.26-000001] failed on step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}]. Moving to ERROR step

GET _ilm/policy/ilm-history-ilm-policy?pretty
{
  "ilm-history-ilm-policy" : {
    "version" : 2,
    "modified_date" : "2022-03-10T03:29:27.915Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "30d"
            }
          }
        }
      }
    },
    "in_use_by" : {
      "indices" : [
        ".ds-ilm-history-5-2021.12.24-000010",
        ".ds-ilm-history-5-2021.11.24-000009",
        ".ds-ilm-history-5-2022.01.23-000011",
        ".ds-ilm-history-5-2022.02.22-000012",
        ".ds-ilm-history-5-2021.10.26-000001"
      ],
      "data_streams" : [
        "ilm-history-5"
      ],
      "composable_templates" : [
        "ilm-history"
      ]
    }
  }
}
2

@wajika You not mentioned which version of ES you are using. But it seems like this is Elasticsearch issue and it fixed in ES 7.14 and ES 8.0 version.

Please check this github PR: 66838 75296

3

I am using 7.15.2.

4

Can you please more log trace. currently it is only one line. it will be great if you can post some above and below log for this erroe line.

5

I don't feel the logs have much valuable content.

[2022-03-10T00:00:05,463][INFO ][o.e.c.m.MetadataCreateIndexService] [es-dev01] [.monitoring-kibana-7-2022.03.10] creating index, cause [auto(bulk api)], templates [.monitoring-kibana], shards [1]/[0]
[2022-03-10T00:00:05,509][INFO ][o.e.c.r.a.AllocationService] [es-dev01] updating number_of_replicas to [1] for indices [.monitoring-kibana-7-2022.03.10]
[2022-03-10T00:00:06,941][INFO ][o.e.c.m.MetadataCreateIndexService] [es-dev01] [.watcher-history-13-2022.03.10] creating index, cause [auto(bulk api)], templates [.watch-history-13], shards [1]/[0]
[2022-03-10T00:00:06,947][INFO ][o.e.c.r.a.AllocationService] [es-dev01] updating number_of_replicas to [1] for indices [.watcher-history-13-2022.03.10]
[2022-03-10T00:00:07,106][INFO ][o.e.x.i.IndexLifecycleTransition] [es-dev01] moving index [.watcher-history-13-2022.03.03] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"delete","action":"delete","name":"wait-for-shard-history-leases"}] in policy [watch-history-ilm-policy]
[2022-03-10T00:00:07,250][INFO ][o.e.c.m.MetadataMappingService] [es-dev01] [.watcher-history-13-2022.03.10/jVw1bVXTQHuQxH3t0x9yVQ] update_mapping [_doc]
[2022-03-10T00:00:07,304][INFO ][o.e.c.m.MetadataMappingService] [es-dev01] [.watcher-history-13-2022.03.10/jVw1bVXTQHuQxH3t0x9yVQ] update_mapping [_doc]
[2022-03-10T00:00:07,360][INFO ][o.e.x.i.IndexLifecycleTransition] [es-dev01] moving index [.watcher-history-13-2022.03.10] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [watch-history-ilm-policy]
[2022-03-10T00:00:08,681][INFO ][o.e.c.m.MetadataCreateIndexService] [es-dev01] [.monitoring-es-7-2022.03.10] creating index, cause [auto(bulk api)], templates [.monitoring-es], shards [1]/[0]
[2022-03-10T00:00:08,691][INFO ][o.e.c.r.a.AllocationService] [es-dev01] updating number_of_replicas to [1] for indices [.monitoring-es-7-2022.03.10]
[2022-03-10T00:02:00,969][INFO ][o.e.x.i.IndexLifecycleTransition] [es-dev01] moving index [.watcher-history-13-2022.03.03] from [{"phase":"delete","action":"delete","name":"wait-for-shard-history-leases"}] to [{"phase":"delete","action":"delete","name":"cleanup-snapshot"}] in policy [watch-history-ilm-policy]
[2022-03-10T00:02:00,972][INFO ][o.e.x.i.IndexLifecycleRunner] [es-dev01] policy [ilm-history-ilm-policy] for index [.ds-ilm-history-5-2021.10.26-000001] on an error step due to a transient error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [9683]
[2022-03-10T00:02:01,082][INFO ][o.e.x.i.IndexLifecycleTransition] [es-dev01] moving index [.watcher-history-13-2022.03.03] from [{"phase":"delete","action":"delete","name":"cleanup-snapshot"}] to [{"phase":"delete","action":"delete","name":"delete"}] in policy [watch-history-ilm-policy]
[2022-03-10T00:02:01,123][INFO ][o.e.c.m.MetadataDeleteIndexService] [es-dev01] [.watcher-history-13-2022.03.03/Y7w3OIk1TmKv0tmTvLjElA] deleting index
[2022-03-10T00:12:00,970][ERROR][o.e.x.i.IndexLifecycleRunner] [es-dev01] policy [ilm-history-ilm-policy] for index [.ds-ilm-history-5-2021.10.26-000001] failed on step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}]. Moving to ERROR step
java.lang.IllegalArgumentException: setting [index.lifecycle.rollover_alias] for index [.ds-ilm-history-5-2021.10.26-000001] is empty or not defined
	at org.elasticsearch.xpack.core.ilm.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:77) [x-pack-core-7.15.2.jar:7.15.2]
	at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:176) [x-pack-ilm-7.15.2.jar:7.15.2]
	at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:365) [x-pack-ilm-7.15.2.jar:7.15.2]
	at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggered(IndexLifecycleService.java:303) [x-pack-ilm-7.15.2.jar:7.15.2]
	at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:184) [x-pack-core-7.15.2.jar:7.15.2]
	at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:217) [x-pack-core-7.15.2.jar:7.15.2]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-03-10T00:22:00,970][INFO ][o.e.x.i.IndexLifecycleRunner] [es-dev01] policy [ilm-history-ilm-policy] for index [.ds-ilm-history-5-2021.10.26-000001] on an error step due to a transient error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [9684]
[2022-03-10T00:32:00,971][ERROR][o.e.x.i.IndexLifecycleRunner] [es-dev01] policy [ilm-history-ilm-policy] for index [.ds-ilm-history-5-2021.10.26-000001] failed on step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}]. Moving to ERROR step
java.lang.IllegalArgumentException: setting [index.lifecycle.rollover_alias] for index [.ds-ilm-history-5-2021.10.26-000001] is empty or not defined
	at org.elasticsearch.xpack.core.ilm.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:77) [x-pack-core-7.15.2.jar:7.15.2]
	at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:176) [x-pack-ilm-7.15.2.jar:7.15.2]
	at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:365) [x-pack-ilm-7.15.2.jar:7.15.2]
	at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggered(IndexLifecycleService.java:303) [x-pack-ilm-7.15.2.jar:7.15.2]
	at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:184) [x-pack-core-7.15.2.jar:7.15.2]
	at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:217) [x-pack-core-7.15.2.jar:7.15.2]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-03-10T00:42:00,971][INFO ][o.e.x.i.IndexLifecycleRunner] [es-dev01] policy [ilm-history-ilm-policy] for index [.ds-ilm-history-5-2021.10.26-000001] on an error step due to a transient error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [9685]
[2022-03-10T00:44:00,010][INFO ][o.e.x.m.MlDailyMaintenanceService] [es-dev01] triggering scheduled [ML] maintenance tasks
[2022-03-10T00:44:00,011][INFO ][o.e.x.m.a.TransportDeleteExpiredDataAction] [es-dev01] Deleting expired data
[2022-03-10T00:44:00,025][INFO ][o.e.x.m.j.r.UnusedStatsRemover] [es-dev01] Successfully deleted [0] unused stats documents
[2022-03-10T00:44:00,026][INFO ][o.e.x.m.a.TransportDeleteExpiredDataAction] [es-dev01] Completed deletion of expired ML data
[2022-03-10T00:44:00,026][INFO ][o.e.x.m.MlDailyMaintenanceService] [es-dev01] Successfully completed [ML] maintenance task: triggerDeleteExpiredDataTask
[2022-03-10T00:52:00,970][ERROR][o.e.x.i.IndexLifecycleRunner] [es-dev01] policy [ilm-history-ilm-policy] for index [.ds-ilm-history-5-2021.10.26-000001] failed on step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}]. Moving to ERROR step
java.lang.IllegalArgumentException: setting [index.lifecycle.rollover_alias] for index [.ds-ilm-history-5-2021.10.26-000001] is empty or not defined
	at org.elasticsearch.xpack.core.ilm.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:77) [x-pack-core-7.15.2.jar:7.15.2]
	at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:176) [x-pack-ilm-7.15.2.jar:7.15.2]
	at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:365) [x-pack-ilm-7.15.2.jar:7.15.2]
	at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggered(IndexLifecycleService.java:303) [x-pack-ilm-7.15.2.jar:7.15.2]
	at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:184) [x-pack-core-7.15.2.jar:7.15.2]
	at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:217) [x-pack-core-7.15.2.jar:7.15.2]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.