Currently i have logs from 5 to 6 servers coming to elk. I want to delete all indices older than a week. what is the best way to achieve this? i went through ILM but didn't get the exact way to do this
ILM can definitely do that - Delete | Elasticsearch Guide [8.5] | Elastic
Are you generating daily indices or one index , If daily indices disable rollover in hot phase and put 7 days at delete phase. if one index , try using rollover after 7 days in hot and put 0 days at delete phase.
For more info:
1 Like
The best way and easy way to do it via the Kibana UI.
You can go to Stack Management --> Index Lifecycle Management --> Create Policy and here in the Hot phase, set the number of days you want a particular index to be in Elasticsearch and then remove the Cold phase.
Automatically after 7 days the data will be deleted.
When i select ILm and create a new policy i am only getting hot, warm and cold phase. no delete phase. Indices are created on a daily basis index_name-@timestamp i want to delete all indices after 7 days
In hot phase, you'll expand in advance settings, you'll find infinite and delete symbol on your right.
Change from infinite to delete.
The delete phase will appear.
1 Like
Yes, it'll.
Click on Advanced settings in hot phase , turn off enable rollover then data will be deleted after 2 days if you put 2 days in delete phase
Ok. So currently in the policy ive selected delete this index after this phase and disabled the enable rollover. and in delete phase i've set it to two days. So i have a index say index_name-@timestamp with timestamp of 5 days old. will it delete all indices older than 2 days?
But daily a new index is getting created with the date. How can i attach a policy to this index? i mean which index should i apply this policy on
All these indexes will have an alias name, and using that it identifies the indices.
how can i give the alias name?
please go through the URL which @Venkata_Raja has given.
It tells you everything which is required.
Thanks.
I went through the doc u shared. but i am not getting anything. I've created a policy as u said. But i am confused as to how to create a template, alias and al.
Alias was not needed in your case i guess as you are disabling rollover. Just create an Index template with the exact pattern that will match all your indices and attach ILM policy to that index as shown in that link.
ok. I'm stuck in creating index template only. i selected create new template. as there was no existing template for all my indices
skipped all the other steps after entering the index pattern.
created policy. created template that matches all the indices, with no other settings. mapped the policy to the index which has timestamp attached to it? how to attach it to the template?
I created the index template with these settings
{
"index": {
"lifecycle": {
"name": "indices-10"
},
"number_of_shards": "1",
"number_of_replicas": "1"
}
}
indices-10 policy is
PUT _ilm/policy/indices-10
{
"policy": {
"phases": {
"hot": {
"min_age": "0ms",
"actions": {
"set_priority": {
"priority": 100
}
}
},
"delete": {
"min_age": "2d",
"actions": {
"delete": {
"delete_searchable_snapshot": true
}
}
}
}
}
}
Do i need to make any changes to logstash config? how will this start working? how can i test it or apply it?
logstash config
input {
beats {
port => 5044
}
}
output {
elasticsearch {
hosts => "http://*.*.*.*:9200"
index => "%{type}%{+YYYY.MM.dd}"
user => "e***"
password => "*****"
}
}