Hello. On ELK stack 7.6.2, I have a policy set for deleting all indices older than 30 days.
The policy was set as follows:
PUT _ilm/policy/cleanup-history
{
"policy": {
"phases": {
"hot": {
"min_age": "0ms",
"actions": {}
},
"delete": {
"min_age": "30d",
"actions": {
"delete": {}
}
}
}
}
}
Out of the indices that are being deleted every 30 days, I want to selectively delete an index e.g beginning with uat_sd_taction*
every 5 days as it tends to grow big very fast.
Please guide me how can I set that option?
Thanks
jportner
(Joe Portner)
February 23, 2021, 9:00pm
2
You'll want to do the following:
Create a second Index Lifecycle Policy to delete an index every 5 days
Create an Index Template to apply your second Index Lifecycle Policy using the index pattern uat_sd_taction*
1 Like
Thanks. So I would presume I could put more than one comma separated index patterns in the template as below. So essentially all the ones that the new policy applies on:
PUT _template/taction_template
{
"index_patterns": ["test-*", "test1-*", "test2-*"],
"settings": {
"number_of_shards": 1,
"number_of_replicas": 1,
"index.lifecycle.name": "my_policy",
"index.lifecycle.rollover_alias": "test-alias"
}
}
Please confirm
jportner
(Joe Portner)
February 24, 2021, 4:11pm
4
Yes, you can specify multiple index patterns like that!
Thanks.
I put an additional policy and template as follows:
PUT _ilm/policy/cleanup-crnlp
{
"policy": {
"phases": {
"hot": {
"min_age": "0ms",
"actions": {}
},
"delete": {
"min_age": "2d",
"actions": {
"delete": {}
}
}
}
}
}
PUT _template/crnlp_template
{
"index_patterns": ["uat_sd_taction1*", "uat_sd_taction2*", "uat_sd_taction3*", "uat_sd_taction4*"],
"settings": {
"number_of_shards": 1,
"number_of_replicas": 1,
"index.lifecycle.name": "cleanup-crnlp",
"index.lifecycle.rollover_alias": "crnlp-alias"
}
}
I chose the index name correctly (followed by an asterisk). For some reason the new policy does not seem linked to any indices and also I don't notice indices older than 2 days getting deleted. See below:
Please advise. Do the old indices start getting deleted right away?
jportner
(Joe Portner)
February 24, 2021, 5:41pm
6
Sorry for the confusion. The index template is only used when new indices are created.
You will need to manually apply your new index lifecycle policy to any existing indices.
See also: Manage existing indices .
Thanks for pointing out the manually applying lifecycle policy to existing indices page. So I created a policy named cleanup-crnlp
as shown above. Now to add existing indices shall I replace test-index
below with my index name? Also my indices are rolling with date and their pattern has names such as uat_sd_taction1*
PUT test-index
{
"settings": {
"number_of_shards": 1,
"number_of_replicas": 1,
"index.lifecycle.name": "cleanup-crnlp"
}
}
In a nutshell, how do I introduce my existing indices (with pattern as uat_sd_taction1*) to the new policy?
Thanks
jportner
(Joe Portner)
February 24, 2021, 9:06pm
8
This should do it:
PUT uat_sd_taction1*/_settings
{
"index": {
"lifecycle": {
"name": "cleanup-crnlp"
}
}
}
Hi Joe,
Thanks. I applied the following:
PUT uat_sd_taction1*/_settings
{
"index": {
"lifecycle": {
"name": "cleanup-crnlp"
}
}
}
It seems that this took effect for the time being as today I see old (over 2 days old) indices again.
Please advise on how can the individual index removal policy could be made permanent.
Thanks
jportner
(Joe Portner)
March 1, 2021, 5:18pm
10
If I'm understanding you correctly: you have newer indices that do not have this policy applied. Is that correct?
Looking at your template above, I think you are too specific with your index patterns. You probably need to change it to this:
PUT _template/crnlp_template
{
"index_patterns": ["uat_sd_taction*"],
"settings": {
"number_of_shards": 1,
"number_of_replicas": 1,
"index.lifecycle.name": "cleanup-crnlp",
"index.lifecycle.rollover_alias": "crnlp-alias"
}
}
Yes the indices keep rolling with date. An example is:
uat_sd_taction-2021.02.25
uat_sd_taction-2021.02.26
uat_sd_taction-2021.02.27
uat_sd_taction-2021.02.28
uat_sd_taction-2021.03.01
I want to keep the most recent two and delete the rest using the following policy:
PUT _ilm/policy/cleanup-crnlp
{
"policy": {
"phases": {
"hot": {
"min_age": "0ms",
"actions": {}
},
"delete": {
"min_age": "2d",
"actions": {
"delete": {}
}
}
}
}
}
jportner
(Joe Portner)
March 1, 2021, 8:31pm
12
OK, then you need to use the index pattern "uat_sd_taction*"
in your template like I mentioned above. That should behave how you want.
So you need to:
Updated your template accordingly
Manually applied the policy to any existing indices again
In the future, check if new indices are using the correct policy in the future with this command:
GET uat_sd_taction*/_ilm/explain
For more info on lifecycle progress see the docs .
Thanks for the help.
I still see issues and see old indices present when I run the following:
GET uat_sd_taction*/_ilm/explain
So far I did the following:
Created the following policy:
UT _ilm/policy/cleanup-crnlp
{
"policy": {
"phases": {
"hot": {
"min_age": "0ms",
"actions": {}
},
"delete": {
"min_age": "2d",
"actions": {
"delete": {}
}
}
}
}
}
The template I created (which employs this policy) is as follows:
PUT _template/2days_crnlp_template
{
"index_patterns": ["uat_sd_taction*", "uat_md_taction*", "uat_fd_taction*", "uat_cd_taction*", "uat_dd_taction*"],
"settings": {
"number_of_shards": 1,
"number_of_replicas": 1,
"index.lifecycle.name": "cleanup-crnlp",
"index.lifecycle.rollover_alias": "crnlp-alias"
}
}
Despite of this in place for a few days, I still see indices "older" than 2 days existing in elasticsearch.
Please guide.
dakrone
(Lee Hinman)
March 24, 2021, 2:19pm
14
Hi there,
It looks like you might have multiple templates that have different lifecycles, if the lifecycle is indeed not being set correctly.
Could you provide the output of GET /uat_sd_taction<something>/_settings
where the index name is one of the old indices that you expect to be deleted (I'd like to see whether the policy was actually set in the index settings).
Also, could you show us the explain output from one of the older indices you expect to be deleted?
system
(system)
Closed
April 21, 2021, 2:20pm
15
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.