ILM delete all indices in X days

Elastic Stack Elasticsearch
1

Hello,

I need some help with understanding ILM configuration.

Here is my configuration for ILM

PUT _ilm/policy/example
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_size": "40gb",
            "max_age": "7d"
          }
        }
      },
      "warm": {
        "actions": {
          "forcemerge": {
            "max_num_segments": 1
          },
          "migrate": {
            "enabled": false
          },
          "shrink": {
            "number_of_shards": 1
          }
        }
      },
      "delete": {
        "min_age": "14d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}

I want to keep all indices max 14 days. Problem is that time for deleting is counted from rollover, which can be processed in 1-7 days. So there are some indices which can be deleted after 14 days (1st day rollover, then 14 days to delete phase), but there are many which can be deleted after 21 days (rollover after 7 days + 14 days to phase age).

Curator was the first idea which came to my mind, I could omit the delete phase and use curator to delete index older than 14 days.

My second idea is that I have a wrong settings on ILM, but I cannot figure how to do it. Is there a posibility to achive what I just described?

2

Your retention period is very short compared to the max period each index covers. If you are keeping 14 days of data I would rollover after a day or two.

3

That has its purpose. There are some indices, which grows slowly and some which grows very quickly. If I had 1 day rollover, there will be many small indices (hundreds of kb size), which is not good for ES as i read in the docs. 7 days rollover means 7 times less these small indices. Sounds reasonable to you?

4

It does, but that is also what limits the accuracy of deletion.

5

Thanks for using Elasticsearch and ILM.

Phase transition timings can be calculated based on the index creation time (instead of rollover time) if the origination date is configured for the managed indices (as described here )

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.