Hello Elastic
We are ingesting a lot of data into Elastic.
We are using index templates.
But our servers are filling up. I want to automate the cleaning of old data, instead of having to manually watch when they're filling up, and then manually delete old indices.
I want to delete data after 30 days.
I think this is done via ILM.
I have been reading the docs about ILM and Index Management for 5 hours now, but it is sounding like this: Technical Jargon Overload - YouTube and the tutorials feel like this: https://i.imgur.com/d0dly15.png
It feels like the docs assume I am already an elastic expert, and know how every intricacy of how the system works, other than what I am currently reading about. It also seems like the docs assume that we use Data Streams, or that we haven't ingested any data yet. And if we have, then it assumes we know how to do a "re-indexing", which sounds very complicated.
I simply want to delete old data. It all feels very complicated for something that seems like it should be simple.
I have an "ILM" policy called "metricbeat", which only has a hot phase with defaults and "Delete data after this phase" selected.
I have a Legacy index template called "metricbeat-7.13.0" with index pattern "metricbeat-7.13.0-*" and ILM policy "metricbeat".
The same is the case for filebeat.
Why do I still have indexes that are more than 30 days old? Shouldn't they be deleted by the policy? How do I debug this and how do I fix it?
For example, why does the 09.30 index still exist, even though it is older than 30 days?
Screenshot is from my Vagrant test environment.
I am getting the follow stack trace when looking at indices with lifecycle errors under Index Management in Kibana.
java.lang.IllegalArgumentException: index.lifecycle.rollover_alias [metricbeat-7.13.0] does not point to index [metricbeat-7.13.0-2021.10.29]
at org.elasticsearch.xpack.core.ilm.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:126)
at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:176)
at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:333)
at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggered(IndexLifecycleService.java:271)
at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:184)
at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:217)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
at java.base/java.lang.Thread.run(Thread.java:831)
Most of the official Documentation seems to be talking about "Data Streams". But we are using Index Templates. How do I make sure data gets deleted after 30 days, so our servers Elastic servers don't get too filled with data?
The beats are going through Logstash.
The Logstash output is looking like this:
output {
if [@metadata][pipeline] {
elasticsearch {
ecs_compatibility => "v1"
pipeline => "%{[@metadata][pipeline]}"
ssl => true
cacert => "/etc/pki/ca-trust/source/anchors/ca.crt"
hosts => [ "es01.sanitized:9200", "es02.sanitized:9200" ]
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
user => "system_logstash_test"
password => "sanitized"
ilm_enabled => "true"
ilm_pattern => "{now/d}-000001"
ilm_policy => "%{[@metadata][beat]}"
ilm_rollover_alias => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
manage_template => false
}
} else {
elasticsearch {
ecs_compatibility => "v1"
ssl => true
cacert => "/etc/pki/ca-trust/source/anchors/_ca.crt"
hosts => [ "es01.sanitized:9200", "es02.sanitized:9200" ]
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
user => "system_logstash_test"
password => "sanitized"
ilm_enabled => "true"
ilm_pattern => "{now/d}-000001"
ilm_policy => "%{[@metadata][beat]}"
ilm_rollover_alias => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
manage_template => false
}
}
}
Trying to add the Policy I get the following warning. What does it mean? Do I really have to manually add an alias for every single day and beat that I want to enroll in ILM?
