ILM - index already exists error

[ollyw (Oliver Whiteman)]

I am new to the ILM side of things and have got over a number of hurdles but now stuck!

I have a set of indexes which include the date in their name e.g. logs-20190708, logs-20190709

My ILM policy has a rollover_alias of the form rollover-logs-20190708, rollover-logs-20190709, etc

The error I get is that when e.g. logs-20190708 goes into the check-rollover-ready phase I get an error saying logs-20190709/blah ResourceAlreadyExistsException. So an index for the following date

Any ideas what is causing this and how to resolve?

[antcas]

In our case, we are using the alias without the date, but we added the date as a formula to the index name when we created it.

For example, we have an index logs-debug-2019.08.05-000156 with the alias logs-debug and when it is rolled over, the new name will be logs-debug-DATE-000157.

Here's some docs on this: https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-rollover-index.html#_using_date_math_with_the_rollover_api

[dakrone (Lee Hinman)]

Hi Oliver,

Can you post the output of

GET /<your-index>/_ilm/explain?human

As well as a copy of the ILM policy so we can try to figure out what's going on?

Thanks!

[ollyw (Oliver Whiteman)]

Apologies been on holiday!

So here is the output from the ilm GET:

{
  "indices" : {
    "xxx-prod-20190803" : {
      "index" : "xxx-prod-20190803",
      "managed" : true,
      "policy" : "xxx-logs-rollover",
      "lifecycle_date" : "2019-08-03T00:00:00.137Z",
      "lifecycle_date_millis" : 1564790400137,
      "phase" : "hot",
      "phase_time" : "2019-08-03T00:00:03.769Z",
      "phase_time_millis" : 1564790403769,
      "action" : "rollover",
      "action_time" : "2019-08-03T00:04:52.406Z",
      "action_time_millis" : 1564790692406,
      "step" : "ERROR",
      "step_time" : "2019-08-03T00:14:51.243Z",
      "step_time_millis" : 1564791291243,
      "failed_step" : "check-rollover-ready",
      "step_info" : {
        "type" : "illegal_argument_exception",
        "reason" : "setting [index.lifecycle.rollover_alias] for index [xxx-prod-20190803] is empty or not defined",
        "stack_trace" : """
java.lang.IllegalArgumentException: setting [index.lifecycle.rollover_alias] for index [kwiff-prod-20190803] is empty or not defined
	at org.elasticsearch.xpack.core.indexlifecycle.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:50)
	at org.elasticsearch.xpack.indexlifecycle.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:133)
	at org.elasticsearch.xpack.indexlifecycle.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:270)
	at org.elasticsearch.xpack.indexlifecycle.IndexLifecycleService.triggered(IndexLifecycleService.java:213)
	at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:168)
	at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:196)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

"""
      },
      "phase_execution" : {
        "policy" : "xxx-logs-rollover",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "7d"
            },
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "version" : 1,
        "modified_date" : "2019-07-04T12:30:18.743Z",
        "modified_date_in_millis" : 1562243418743
      }
    }
  }
}

The policy has very little in it - was created using the GUI:

{
  "xxx-logs-rollover" : {
    "version" : 1,
    "modified_date" : "2019-07-04T12:30:18.743Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "7d"
            },
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "warm" : {
          "min_age" : "14d",
          "actions" : {
            "allocate" : {
              "include" : { },
              "exclude" : { },
              "require" : {
                "data" : "warm"
              }
            },
            "forcemerge" : {
              "max_num_segments" : 1
            },
            "set_priority" : {
              "priority" : 50
            },
            "shrink" : {
              "number_of_shards" : 1
            }
          }
        },
        "cold" : {
          "min_age" : "30d",
          "actions" : {
            "set_priority" : {
              "priority" : 0
            }
          }
        }
      }
    }
  }
}