Hi
I am using ELK stack 7.2, I have an ILM policy set but looks like its not working. The maximum index size I set is 14GB.
Note: I am running ES on a single vm.
I see the index size is 28gb but it does not rollover to a new index.
Can you run, via the console:
GET metricbeat-2019.06.25-000001/_ilm/explain
And post the output here? This will tell us what ILM thinks is going on with this index.
{
"indices" : {
"metricbeat-2019.06.25-000001" : {
"index" : "metricbeat-2019.06.25-000001",
"managed" : true,
"policy" : "metricbeat-7.2.0",
"lifecycle_date_millis" : 1561497717119,
"phase" : "hot",
"phase_time_millis" : 1562707292481,
"action" : "rollover",
"action_time_millis" : 1562707378617,
"step" : "ERROR",
"step_time_millis" : 1562707978577,
"failed_step" : "check-rollover-ready",
"step_info" : {
"type" : "illegal_argument_exception",
"reason" : "index [metricbeat] is not the write index for alias [metricbeat-2019.06.25-000001]",
"stack_trace" : """
java.lang.IllegalArgumentException: index [metricbeat] is not the write index for alias [metricbeat-2019.06.25-000001]
at org.elasticsearch.xpack.core.indexlifecycle.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:100)
at org.elasticsearch.xpack.indexlifecycle.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:133)
at org.elasticsearch.xpack.indexlifecycle.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:270)
at org.elasticsearch.xpack.indexlifecycle.IndexLifecycleService.triggered(IndexLifecycleService.java:213)
at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:168)
at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:196)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"""
},
"phase_execution" : {
"policy" : "metricbeat-7.2.0",
"phase_definition" : {
"min_age" : "0ms",
"actions" : {
"rollover" : {
"max_size" : "14gb",
"max_age" : "5d"
}
}
},
"version" : 48,
"modified_date_in_millis" : 1562704667671
}
}
}
}Currently I have these indicex.
metricbeat-2019.06.25-00001 was created after roll over.
What I have noticed is that the ILM policy is being over written after installing a new metricbeats agent on servers with every install.
metricbeat-2019.06.25-00001 -- This rollover index was created after removing the index from ILM policy and then adding it back.
One more thing. When ever I install the beats systems it overwrites the ilm policy.
But when I look at the index it does not take that new policy. As you see in the image its showing the old ilm policy. For index to use the new policy I have to remove and add the ILM policy back to that index.
That error message indicates (although there's a bug in the error message, the index name and the alias name should be swapped) that there isn't an alias metricbeat that points to the index metricbeat-2019.06.25-000001. If you've confirmed that the alias exists, and metricbeat-2019.06.25-000001 has is_write_index: true for that alias, try using the Retry policy API.
You may be better off asking about this in the Beats section of the forum, but it looks like you can prevent new Beats from doing this by setting setup.ilm.overwrite to false in the Beats configuration (see the ILM settings section of the Beats documentation).
I tried this option. setup.ilm.overwrite to false
But this does not work.
Unfortunately I'm not a Beats expert so I can't really troubleshoot that very effectively - I recommend opening a new topic in the Beats section of this forum.
Thank you
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.