ILM - index already exists error

I am new to the ILM side of things and have got over a number of hurdles but now stuck!

I have a set of indexes which include the date in their name e.g. logs-20190708, logs-20190709

My ILM policy has a rollover_alias of the form rollover-logs-20190708, rollover-logs-20190709, etc

The error I get is that when e.g. logs-20190708 goes into the check-rollover-ready phase I get an error saying logs-20190709/blah ResourceAlreadyExistsException. So an index for the following date

Any ideas what is causing this and how to resolve?

In our case, we are using the alias without the date, but we added the date as a formula to the index name when we created it.

For example, we have an index logs-debug-2019.08.05-000156 with the alias logs-debug and when it is rolled over, the new name will be logs-debug-DATE-000157.

Here's some docs on this: https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-rollover-index.html#_using_date_math_with_the_rollover_api

Hi Oliver,

Can you post the output of

GET /<your-index>/_ilm/explain?human

As well as a copy of the ILM policy so we can try to figure out what's going on?

Thanks!

Apologies been on holiday!

So here is the output from the ilm GET:

{
  "indices" : {
    "xxx-prod-20190803" : {
      "index" : "xxx-prod-20190803",
      "managed" : true,
      "policy" : "xxx-logs-rollover",
      "lifecycle_date" : "2019-08-03T00:00:00.137Z",
      "lifecycle_date_millis" : 1564790400137,
      "phase" : "hot",
      "phase_time" : "2019-08-03T00:00:03.769Z",
      "phase_time_millis" : 1564790403769,
      "action" : "rollover",
      "action_time" : "2019-08-03T00:04:52.406Z",
      "action_time_millis" : 1564790692406,
      "step" : "ERROR",
      "step_time" : "2019-08-03T00:14:51.243Z",
      "step_time_millis" : 1564791291243,
      "failed_step" : "check-rollover-ready",
      "step_info" : {
        "type" : "illegal_argument_exception",
        "reason" : "setting [index.lifecycle.rollover_alias] for index [xxx-prod-20190803] is empty or not defined",
        "stack_trace" : """
java.lang.IllegalArgumentException: setting [index.lifecycle.rollover_alias] for index [kwiff-prod-20190803] is empty or not defined
	at org.elasticsearch.xpack.core.indexlifecycle.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:50)
	at org.elasticsearch.xpack.indexlifecycle.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:133)
	at org.elasticsearch.xpack.indexlifecycle.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:270)
	at org.elasticsearch.xpack.indexlifecycle.IndexLifecycleService.triggered(IndexLifecycleService.java:213)
	at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:168)
	at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:196)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

"""
      },
      "phase_execution" : {
        "policy" : "xxx-logs-rollover",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "7d"
            },
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "version" : 1,
        "modified_date" : "2019-07-04T12:30:18.743Z",
        "modified_date_in_millis" : 1562243418743
      }
    }
  }
}

The policy has very little in it - was created using the GUI:

{
  "xxx-logs-rollover" : {
    "version" : 1,
    "modified_date" : "2019-07-04T12:30:18.743Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "7d"
            },
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "warm" : {
          "min_age" : "14d",
          "actions" : {
            "allocate" : {
              "include" : { },
              "exclude" : { },
              "require" : {
                "data" : "warm"
              }
            },
            "forcemerge" : {
              "max_num_segments" : 1
            },
            "set_priority" : {
              "priority" : 50
            },
            "shrink" : {
              "number_of_shards" : 1
            }
          }
        },
        "cold" : {
          "min_age" : "30d",
          "actions" : {
            "set_priority" : {
              "priority" : 0
            }
          }
        }
      }
    }
  }
}