Creating Index Lifecycle Policies

Elastic Stack Kibana
#1

Hello. On ELK stack 7.6.2, I have a policy set for deleting all indices older than 30 days.

The policy was set as follows:

PUT _ilm/policy/cleanup-history
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {}
      },
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Out of the indices that are being deleted every 30 days, I want to selectively delete an index e.g beginning with uat_sd_taction* every 5 days as it tends to grow big very fast.

Please guide me how can I set that option?

Thanks

#2

You'll want to do the following:

  • Create a second Index Lifecycle Policy to delete an index every 5 days
  • Create an Index Template to apply your second Index Lifecycle Policy using the index pattern uat_sd_taction*
1 Like
#3

Thanks. So I would presume I could put more than one comma separated index patterns in the template as below. So essentially all the ones that the new policy applies on:


PUT _template/taction_template
{
  "index_patterns": ["test-*", "test1-*", "test2-*"], 
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 1,
    "index.lifecycle.name": "my_policy", 
    "index.lifecycle.rollover_alias": "test-alias" 
  }
}

Please confirm

#4

Yes, you can specify multiple index patterns like that! :grinning_face_with_smiling_eyes:

#5

Thanks.

I put an additional policy and template as follows:

PUT _ilm/policy/cleanup-crnlp
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {}
      },
      "delete": {
        "min_age": "2d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}


PUT _template/crnlp_template
{
  "index_patterns": ["uat_sd_taction1*", "uat_sd_taction2*", "uat_sd_taction3*", "uat_sd_taction4*"], 
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 1,
    "index.lifecycle.name": "cleanup-crnlp",
    "index.lifecycle.rollover_alias": "crnlp-alias"
    }
}

I chose the index name correctly (followed by an asterisk). For some reason the new policy does not seem linked to any indices and also I don't notice indices older than 2 days getting deleted. See below:

image
image1199×257 20.2 KB

Please advise. Do the old indices start getting deleted right away?

#6

Sorry for the confusion. The index template is only used when new indices are created.

You will need to manually apply your new index lifecycle policy to any existing indices.

See also: Manage existing indices.

#7

Thanks for pointing out the manually applying lifecycle policy to existing indices page. So I created a policy named cleanup-crnlp as shown above. Now to add existing indices shall I replace test-index below with my index name? Also my indices are rolling with date and their pattern has names such as uat_sd_taction1*

PUT test-index
{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 1,
    "index.lifecycle.name": "cleanup-crnlp"
  }
}

In a nutshell, how do I introduce my existing indices (with pattern as uat_sd_taction1*) to the new policy?

Thanks

#8

This should do it:

PUT uat_sd_taction1*/_settings 
{
  "index": {
    "lifecycle": {
      "name": "cleanup-crnlp"
    }
  }
}
#9

Hi Joe,

Thanks. I applied the following:

PUT uat_sd_taction1*/_settings 
{
  "index": {
    "lifecycle": {
      "name": "cleanup-crnlp"
    }
  }
}

It seems that this took effect for the time being as today I see old (over 2 days old) indices again.

Please advise on how can the individual index removal policy could be made permanent.

Thanks

#10

If I'm understanding you correctly: you have newer indices that do not have this policy applied. Is that correct?

Looking at your template above, I think you are too specific with your index patterns. You probably need to change it to this:

PUT _template/crnlp_template
{
  "index_patterns": ["uat_sd_taction*"], 
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 1,
    "index.lifecycle.name": "cleanup-crnlp",
    "index.lifecycle.rollover_alias": "crnlp-alias"
    }
}
#11

Yes the indices keep rolling with date. An example is:

uat_sd_taction-2021.02.25
uat_sd_taction-2021.02.26
uat_sd_taction-2021.02.27
uat_sd_taction-2021.02.28
uat_sd_taction-2021.03.01

I want to keep the most recent two and delete the rest using the following policy:

PUT _ilm/policy/cleanup-crnlp
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {}
      },
      "delete": {
        "min_age": "2d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}
#12

OK, then you need to use the index pattern "uat_sd_taction*" in your template like I mentioned above. That should behave how you want.

So you need to:

  1. Updated your template accordingly
  2. Manually applied the policy to any existing indices again
  3. In the future, check if new indices are using the correct policy in the future with this command:
GET uat_sd_taction*/_ilm/explain

For more info on lifecycle progress see the docs.

#13

Thanks for the help.

I still see issues and see old indices present when I run the following:

GET uat_sd_taction*/_ilm/explain

So far I did the following:

  1. Created the following policy:
UT _ilm/policy/cleanup-crnlp
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {}
      },
      "delete": {
        "min_age": "2d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

The template I created (which employs this policy) is as follows:

PUT _template/2days_crnlp_template
{
  "index_patterns": ["uat_sd_taction*", "uat_md_taction*", "uat_fd_taction*", "uat_cd_taction*", "uat_dd_taction*"], 
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 1,
    "index.lifecycle.name": "cleanup-crnlp",
    "index.lifecycle.rollover_alias": "crnlp-alias"
    }
}

Despite of this in place for a few days, I still see indices "older" than 2 days existing in elasticsearch.

Please guide.

#14

Hi there,

It looks like you might have multiple templates that have different lifecycles, if the lifecycle is indeed not being set correctly.

Could you provide the output of GET /uat_sd_taction<something>/_settings where the index name is one of the old indices that you expect to be deleted (I'd like to see whether the policy was actually set in the index settings).

Also, could you show us the explain output from one of the older indices you expect to be deleted?

#15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.