Hi,
I have some newbe questions about ILM.
Currently I am rotating indices daily or weekly. Logstash is setting the index where to write with date suffix at the end of the index name. I have a cron job which delete indices per elasticsearch's rest api which are older than x days. The job is running once a day.
So my index deletion is indirectly based on @timestamp. That is what I want. My primary goal is to keep data for 7 days for example, based on @timestamp, not derived from index creation date.
That would be nearly the same if your elastic stack and your log producers are always working and live.
Do I understand ILM correctly, that the deletion phase can be only based on index creation or rollover action?
My concern is the following:
I calculate my cluster to be able to hold data for 7 days plus some buffer if some application is writing more logs than usual on massive error state.
But I am afraid that I need a much bigger buffer if a single log producer, can dump old logs (older than 7 days) to my cluster. With my old daily index approach my rest script would delete indices which holds older than my 7 days once a day.
When ILM is only acting based on index creation time, I may have to hold data of this faulty log producer much more than 7 days which reduces my buffer significantly.
Also I have the target to hold data for some specific time (neither longer, nor shorter). But rollover may be very different when also using rollover by time AND size.
Any idea how to deal with these issues?
Is it possible to combine logstash's daily rotation (index with date suffix) and size based rollover with ilm for example?
Then I could delete daily via rest api and I ILM could prevent to big shards or could do some shrinking or other things ILM has to offer.
Thanks a lot, Andreas