Recommended setup using ILM to implement a document retention of n days

Elastic Stack Elasticsearch
3

@gjelu, thank you very much for your answer. It helped me a lot.
Nothing in my tests made sense to me until I finally re-read the last sentence of what you wrote:

"Check the index setting index.provided_name after filebeat creates the first index."

I didn't get it until I tried out the example of using date math from the docs:
https://www.elastic.co/guide/en/elasticsearch/reference/7.5/indices-rollover-index.html#_using_date_math_with_the_rollover_api

Then I understood what you meant by using a minute-based pattern for testing. In my first attempt I turned on "parse_origination_date" and created an index with the hard-coded name of yesterday's date and checked the result with a dry_run rollover. The parse_origination_date must of cource be turned off in the template when

I used the example from the docs above with your pattern like this:

PUT /%3Clogs-%7Bnow%2Fm%7Byyyy.MM.dd.HH.m%7D%7D-1%3E 
{
  "aliases": {
    "logs_write": {}
  }
}

And then tried the rollover like this:

POST /logs_write/_rollover?dry_run=true 
{
  "conditions": {
    "max_docs":   "1"
  }
}
....
{
  "old_index" : "logs-2020.01.04.13.57-1",
  "new_index" : "logs-2020.01.04.14.13-000002",
}

So today's big learning is that the index.provided_name of the current write index MUST contain some date math in order to make use of the date (and time) part when creating the new index with the rollover feature. You can't create an index with yesterday's date hard coded in the name to test it. Using a minute based pattern is easiest for testing, to avoid the wait for the next date :wink:

I also learned that the rollover API accepts a target index name that is useful for testing different date math patterns, thus a simplification or first having to create an index with the pattern to test defined in its provided_name.

Example using the filebeat default rollover alias:

POST filebeat-7.5.1/_rollover/%3Cfilebeat-7.5.1-%7Bnow%2Fm%7Byyyy.MM.dd.HH.m%7D%7D-1%3E?dry_run=true
{
  "conditions": {
    "max_age": "1d"
  }
}
....
{
  "old_index" : "filebeat-7.5.1-2019.12.27-000011",
  "new_index" : "filebeat-7.5.1-2020.01.04.23.21-1",
}

The reason for my issues is after upgrading ES+Beats to 7.x where the Beats default is to use ILM if it's enabled in ES. The name of the write index in use at the point of the upgrade was not created using dath math, thus the reason for ILM not to change the date part during rollover.

So lesson 2 of today is to check the index settings provided name of the current write index when manually enabling the rollover phase in an ILM policy.