Lifecycle Policy doesn't delete documents correctly

Elastic Stack Elasticsearch
1

Good morning!
I set up a lifecycle policy as follows last week, in the hope that documents older than 3 days would be deleted but it hasn't worked, can you tell me why? My intention is to store the data for only that long, is there any way I could trigger the execution of that policy manually (or delete documents older than X days manually?). Thanks

PUT _ilm/policy/metricbeat
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {}
      },
      "delete": {
        "min_age": "3d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}
2

What are the indices linked to this policy? How long time period does each index cover?

Note that ILM deletes complete indices older than 3 days and not individual documents.

3

Thank you for your quick reply Christian.
In the case of this policy, the only index you have linked is "metricbeat-7.12.1-2021.06.01-000001".
Regarding how much time each index covers where could I check it, do you mean the "Maximum age" value of "Rollover" inside the hot phase?
Thanks for the clarification that ILM removes the entire indexes and not the individual documents, anyway, it is still useful for my use case.

4

If I recall correctly I think timne start to cound once the index rolls over, which does not seem to have happened yet. How come your policy does not specify rollover parameters?

5

I wasn't sure if it could be due to that and I tried to disable the "Rollover" option, however, in other indexes where it's enabled, the index hasn't been removed either.
Taking into account what you said, that the time starts counting once the index is restarted, I'm going to test it by setting the "Maximum age" value to 2 minutes and the "Delete phase" to 5 minutes, so that every 2 minutes a new index is generated and every 5 minutes the previous one is deleted, should it work like that?

6

I believe ILM only checks every 10 minutes or so which can cause problems with unrealistic test settings.

If you generate hourly indices and set the delete phase to 2 hours I think the indices would get deleted once 2 hours have passed since they rolled over.

7

Perfect, I will try those values, thank you very much!

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.