ILM policy is not working

Elastic Stack Elasticsearch
1

Hi

I am using ELK stack 7.2, I have an ILM policy set but looks like its not working. The maximum index size I set is 14GB.
Note: I am running ES on a single vm.

image
image.png1311×702 47.3 KB

I see the index size is 28gb but it does not rollover to a new index.

image
image.png1467×188 19.8 KB

2

Can you run, via the console:

GET metricbeat-2019.06.25-000001/_ilm/explain

And post the output here? This will tell us what ILM thinks is going on with this index.

3 
{
  "indices" : {
    "metricbeat-2019.06.25-000001" : {
      "index" : "metricbeat-2019.06.25-000001",
      "managed" : true,
      "policy" : "metricbeat-7.2.0",
      "lifecycle_date_millis" : 1561497717119,
      "phase" : "hot",
      "phase_time_millis" : 1562707292481,
      "action" : "rollover",
      "action_time_millis" : 1562707378617,
      "step" : "ERROR",
      "step_time_millis" : 1562707978577,
      "failed_step" : "check-rollover-ready",
      "step_info" : {
        "type" : "illegal_argument_exception",
        "reason" : "index [metricbeat] is not the write index for alias [metricbeat-2019.06.25-000001]",
        "stack_trace" : """
java.lang.IllegalArgumentException: index [metricbeat] is not the write index for alias [metricbeat-2019.06.25-000001]
	at org.elasticsearch.xpack.core.indexlifecycle.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:100)
	at org.elasticsearch.xpack.indexlifecycle.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:133)
	at org.elasticsearch.xpack.indexlifecycle.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:270)
	at org.elasticsearch.xpack.indexlifecycle.IndexLifecycleService.triggered(IndexLifecycleService.java:213)
	at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:168)
	at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:196)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"""
      },
      "phase_execution" : {
        "policy" : "metricbeat-7.2.0",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "14gb",
              "max_age" : "5d"
            }
          }
        },
        "version" : 48,
        "modified_date_in_millis" : 1562704667671
      }
    }
  }
}
4

@gbrown

Currently I have these indicex.

image
image.png1405×227 25.5 KB

metricbeat-2019.06.25-00001 was created after roll over.

What I have noticed is that the ILM policy is being over written after installing a new metricbeats agent on servers with every install.

metricbeat-2019.06.25-00001 -- This rollover index was created after removing the index from ILM policy and then adding it back.

5

@gbrown

One more thing. When ever I install the beats systems it overwrites the ilm policy.

But when I look at the index it does not take that new policy. As you see in the image its showing the old ilm policy. For index to use the new policy I have to remove and add the ILM policy back to that index.

image
image.png908×745 63.7 KB

6

That error message indicates (although there's a bug in the error message, the index name and the alias name should be swapped) that there isn't an alias metricbeat that points to the index metricbeat-2019.06.25-000001. If you've confirmed that the alias exists, and metricbeat-2019.06.25-000001 has is_write_index: true for that alias, try using the Retry policy API.

You may be better off asking about this in the Beats section of the forum, but it looks like you can prevent new Beats from doing this by setting setup.ilm.overwrite to false in the Beats configuration (see the ILM settings section of the Beats documentation).

7

I tried this option. setup.ilm.overwrite to false

But this does not work.

8

Unfortunately I'm not a Beats expert so I can't really troubleshoot that very effectively - I recommend opening a new topic in the Beats section of this forum.

9

Thank you

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.