Misunderstood ILM

Elastic Stack Elasticsearch
1

I was looking at a solution to delete old logs and I was suggested to set up ILM.

I only now realize that ILM rolls over the entire index rather than old entries. If I understand correctly if I set up ILM to 3 months and then move to deletion, after 3 months it would delete the entire index.

Is there a way to set it so after 3 months, only entires that are older than 3 months will be deleted? Is it possible to do so for storange as well (once reaches 250GB delete old entires)?

Thanks ahead!

2

You'd need to create a custom process to handle that with a delete-by-query.
It's highly inefficient though, and you'd be better off using ILM and deleting entire indices instead.

1 Like
3

Thanks for the response,

The problem with that is that this would delete indexes with very recent data. I assume there's a best practice I'm missing here.

4

Not sure I follow why that would be the case?

5

Best practice is to use time-based indices where data gets assigned to indices based on when they occur. The oldest indices therefore hold the oldest data and indices are deleted oldest first when all data in them has exceeded the retention period.

1 Like
6

Thanks for the response!

May I ask for example how to set it so that logstash would create a new indice each month?

Right now this is my output in logstash:

index => "%{[@metadata][beat]}-%{[@metadata][version]}"
7

Because as of now, I'm not creating time based indices. all the output goes to one single indice. This means that if it gets 3 months old it would be deleted, even though it recieved a document 5 minutes ago.

Sorry if I'm not clear, this is pretty new to me. Please tell me if I could phrase it better.

8

No worries, thanks for explaining your current approach!

Have a read of this page of the docs, it gives a practical example of how this would work, the overall approach is how it'd work with Logstash as well.

9

Thanks for the response,

I couldn't find logstash output reference in that guide. If in logstash I do:

index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"

and in ILM simply do: Hot for 3 months -> delete. Would this remove day worth of data everyday once it reaches 3 months?

10

Ok try https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-ilm

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.