What is the recommended setup using ILM to implement a document retention of n days?
I'm trying to get ILM working with data retention based on number of days since document @timestamp to offer a log service where n days of log data is available in Kibana.
Pre-ILM, I've used a cron job that curled a DELETE request to the index with the date n days ago.
That solution required the index name (logstash-2019.12.29 for example) to match the documents timestamp, thus requiring the data source to pick the index name based on the document data.
Since ILM came around I could use it to delete indices n days old, which works fine as long as the indices are created on the date in the name.
Now when I upgraded our filebeat agents and ES to 7.5.1 and activated ILM, it started to use index names like filebeat-2019.12.20-000001 and the next day filebeat-2019.12.20-000002 with a rollover set to max_age 1d. This means that the date part isn't changing in the index name.
I also discovered that delete phase min_age now acts on date for rollover and not index creation.
My conclusion is that I should turn off rollover in the ILM policy, but keep the delete phase at age of n days, and make sure our agents continue to use the index pattern filebeat-Y.m.d without ILM enabled in the filebeat config to make this work as per our requirements.
This results in very different sizes of indices from day to day but if you use rollover + delete I can't see how I can provide n days of log data because the delete phase then acts on rollover date (could be 1h after creation some days with max size set) and not creation date.