Getting errors related to ILM

Elastic Stack Elasticsearch
,
1

currently running ES 6.7.1 and we use the logstash-* indexes

I recently setup ILM through the UI, using these settings:

{
"policy": {
"phases": {
"hot": {
"min_age": "0ms",
"actions": {
"rollover": {
"max_age": "15d",
"max_size": "50gb"
},
"set_priority": {
"priority": 100
}
}
},
"delete": {
"min_age": "1d",
"actions": {
"delete": {}
}
}
}
}
}

And I noticed in our elasticsearch logs we got the following error:

[2019-05-22T00:16:46,994][ERROR][o.e.x.i.IndexLifecycleRunner] [qmfFc9w] policy [15-days] for index [logstash-2019.05.22] failed on step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}]. Moving to ERROR step
java.lang.IllegalArgumentException: setting [index.lifecycle.rollover_alias] for index [logstash-2019.05.22] is empty or not defined
at org.elasticsearch.xpack.core.indexlifecycle.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:50) [x-pack-core-6.7.0.jar:6.7.0]
at org.elasticsearch.xpack.indexlifecycle.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:133) [x-pack-ilm-6.7.0.jar:6.7.0]
at org.elasticsearch.xpack.indexlifecycle.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:250) [x-pack-ilm-6.7.0.jar:6.7.0]
at org.elasticsearch.xpack.indexlifecycle.IndexLifecycleService.triggered(IndexLifecycleService.java:205) [x-pack-ilm-6.7.0.jar:6.7.0]
at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:168) [x-pack-core-6.7.0.jar:6.7.0]
at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:196) [x-pack-core-6.7.0.jar:6.7.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_201]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_201]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_201]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_201]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_201]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_201]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]

I think I need to set the index.lifecycle.rollover_alias using something like the instructions here:

but it's not totally clear to me what the alias should be named. Any help would be much appreciated.

2

Hi Daniel,

So since you are using rollover, ILM needs to know what alias to do the swap on when the new index is created. This is set in the index settings (or in the template, more likely), for example:

PUT _template/ls-template
{
  "index_patterns": ["logstash-*"],
  "settings": {
    "index.lifecycle.name": "your-policy",
    "index.lifecycle.rollover_alias": "logstash"
  }
}

So any index that is created starting with logstash- will automatically use your policy.

Next, you'll need to create the very first index and the alias, to kick things off:

PUT /logstash-000001
{
  "aliases": {
    "logstash":{
      "is_write_index": true
    }
  }
}

This creates the logstash-000001 index and also creates an alias called "logstash" that points to this new index. The ILM policy can now use this alias for swapping during the rollover.

Hope this helps!

3

Thanks @dakrone. One question though, I think (or thought) that I had done this through the UI already. If I view my ILM policies, then click on Actions -> Add policy to index template this sounds like it should do the same thing, would it not? I did do this through the UI. Is there a way to view the current settings and see if this is already setup?

Or maybe I'm confused about what this action in the UI does.

1 Like
4

I made the changes that were suggested above and now I'm getting the following errors in my logstash.log file:

[2019-05-30T00:16:46,994][ERROR][o.e.x.i.IndexLifecycleRunner] [qmfFc9w] policy [15-days] for 
index [logstash-2019.05.30] failed on step [{"phase":"hot","action":"rollover","name":"check- rollover-ready"}]. Moving to ERROR step
java.lang.IllegalArgumentException: index.lifecycle.rollover_alias [logstash] does not point to index [logstash-2019.05.30]
at org.elasticsearch.xpack.core.indexlifecycle.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:92) [x-pack-core-6.7.0.jar:6.7.0]
at org.elasticsearch.xpack.indexlifecycle.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:133) [x-pack-ilm-6.7.0.jar:6.7.0]
at org.elasticsearch.xpack.indexlifecycle.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:250) [x-pack-ilm-6.7.0.jar:6.7.0]
at org.elasticsearch.xpack.indexlifecycle.IndexLifecycleService.triggered(IndexLifecycleService.java:205) [x-pack-ilm-6.7.0.jar:6.7.0]
at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:168) [x-pack-core-6.7.0.jar:6.7.0]
at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:196) [x-pack-core-6.7.0.jar:6.7.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_201]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_201]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_201]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_201]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_201]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_201]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]

I ran the commands almost exactly as shown above by @dakrone, the only thing I changed was "your-policy" to "15-day" which is what I named the policy in the UI when I created it. Perhaps index.lifecycle.rollover_alias needs to be "logstash-*" instead of "logstash"?

5

Would anyone be able to share with me how they have their index.lifecycle.rollover_alias set to? I'd really like to get this working, seems like I'm close and just barely off here.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.