Delete Index after x amount of days

Elastic Stack Elasticsearch
1

Currently reading logs from Logstash to Elasticsearch and creating daily indexes. Is there a way to delete each of these indexes after an X amount of days?
Also, is there an auto delete feature when my elasticsearch cluster runs out of storage?

2

ILM is the best thing for what you want.

3

I've created an Index Lifecycle Policy but my daily log indexes don't show up when I try to apply the policy to an index template. Is there an extra step I'm missing?

4

Can you share your policy and index template where that's been applied?

5

Screen Shot 2020-08-26 at 5.29.26 PM
Screen Shot 2020-08-26 at 5.29.26 PM2284×474 42.2 KB
Screen Shot 2020-08-26 at 5.30.06 PM
Screen Shot 2020-08-26 at 5.30.06 PM970×1366 38.6 KB

Here are my settings for the Policy and template. What I want the policy to do is to delete the daily indexes after two weeks. My current set up, I believe, will delete all the indexes after two weeks and then start repopulating.

My indexes are set up like this: "jira-year.month.day"

6

You might need to check with the explain API to see what's happening.

7

{
"indices" : {
"jiralogdata-2020.08.21" : {
"index" : "jiralogdata-2020.08.21",
"managed" : false
}
}
}
Looks like the policy isn't being applied, not sure why

8

Thanks! Can you try https://www.elastic.co/guide/en/elasticsearch/reference/7.9/set-up-lifecycle-policy.html#apply-policy-multiple

9

Great! That seemed to add the policy. Is there a way get when the log will be deleted? I'm only getting the information below:

"indices" : {
    "jiralogdata-2020.08.22" : {
      "index" : "jiralogdata-2020.08.22",
      "managed" : true,
      "policy" : "Delete_Index_After_2_weeks",
      "lifecycle_date_millis" : 1598054395357,
      "age" : "5.05d",
      "phase" : "hot",
      "phase_time_millis" : 1598491271739,
      "action" : "complete",
      "action_time_millis" : 1598491272042,
      "step" : "complete",
      "step_time_millis" : 1598491272042,
      "phase_execution" : {
        "policy" : "Delete_Index_After_2_weeks",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "set_priority" : {
              "priority" : 100
            }
          }
        },
1 Like
10

By default, the cluster will check if there are ILM policies it needs to execute.

So you should see something happening after 20 min at the most.

11

Great thanks for your help!

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.