# My lifecycle policy is not working **URL:** \ **Category:** Elasticsearch\ **Tags:** ilm-index-lifecycle-management\ **Created:** [May 9, 2023, 3:00am UTC](https://discuss.elastic.co/t/my-lifecycle-policy-is-not-working/332856 "2023-05-09T03:00:03Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![its-ogawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/its-ogawa/32/120829_2.png) [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)\ **Post date:** [May 9, 2023, 3:00am UTC](https://discuss.elastic.co/t/my-lifecycle-policy-is-not-working/332856/1 "2023-05-09T03:00:03Z")
I want to automatically delete indexes that are 60 days old and have set up the following lifecycle policy. ```auto # curl -XGET '_ilm/policy/its_index-policy?pretty' { "its_index-policy" : { "version" : 1, "modified_date" : "2023-02-21T08:37:18.466Z", "policy" : { "phases" : { "warm" : { "min_age" : "5d", "actions" : { } }, "hot" : { "min_age" : "0ms", "actions" : { "rollover" : { "max_size" : "20gb", "max_age" : "2d", "max_docs" : 50000000 } } }, "delete" : { "min_age" : "60d", "actions" : { "delete" : { "delete_searchable_snapshot" : true } } } } }, "in_use_by" : { "indices" : [ ... snip ... ], "data_streams" : [], "composable_templates" : [ "its_index_template" ] } } } ``` However, there are still indexes that are 60 days old. ```auto # curl -XGET 'example-2023.02.22/_ilm/explain?pretty' { "indices" : { "example-2023.02.22" : { "index" : "example-2023.02.22", "managed" : true, "policy" : "its_index-policy", "index_creation_date_millis" : 1677054544127, "time_since_index_creation" : "75.75d", "lifecycle_date_millis" : 1677054544127, "age" : "75.75d", "phase" : "hot", "phase_time_millis" : 1683599336458, "action" : "rollover", "action_time_millis" : 1677054544636, "step" : "ERROR", "step_time_millis" : 1683599936449, "failed_step" : "check-rollover-ready", "is_auto_retryable_error" : true, "failed_step_retry_count" : 5454, "step_info" : { "type" : "illegal_argument_exception", "reason" : "index.lifecycle.rollover_alias [its_index-policy-alias] does not point to index [example-2023.02.22]" }, "phase_execution" : { "policy" : "its_index-policy", "phase_definition" : { "min_age" : "0ms", "actions" : { "rollover" : { "max_size" : "20gb", "max_age" : "2d", "max_docs" : 50000000 } } }, "version" : 1, "modified_date_in_millis" : 1676968638466 } } } } ``` What am I doing wrong? ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [May 9, 2023, 3:21am UTC](https://discuss.elastic.co/t/my-lifecycle-policy-is-not-working/332856/2 "2023-05-09T03:21:48Z")
What version are you on? What are all the steps you followed to get to this point? The error states ```auto "reason" : "index.lifecycle.rollover_alias [its_index-policy-alias] does not point to index [example-2023.02.22]" ``` Your writes alias is not correct so the index can not rollover... no rollover not moving on to the next phase. What does this show `GET its_index-policy-alias` ---
**Author:** ![its-ogawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/its-ogawa/32/120829_2.png) [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)\ **Post date:** [May 9, 2023, 3:29am UTC](https://discuss.elastic.co/t/my-lifecycle-policy-is-not-working/332856/3 "2023-05-09T03:29:47Z")
Thank you for your prompt response. The version of elasticsearch is 8.6.2. ```auto { "name" : "ITS-ELST-01", "cluster_name" : "els-cluster", "cluster_uuid" : "VhwbgOduQlqwvUB_1t1HiA", "version" : { "number" : "8.6.2", "build_flavor" : "default", "build_type" : "rpm", "build_hash" : "2d58d0f136141f03239816a4e360a8d17b6d8f29", "build_date" : "2023-02-13T09:35:20.314882762Z", "build_snapshot" : false, "lucene_version" : "9.4.2", "minimum_wire_compatibility_version" : "7.17.0", "minimum_index_compatibility_version" : "7.0.0" }, "tagline" : "You Know, for Search" } ``` The circumstances that led me to this problem were that I encountered the following logs. ```auto # tail /var/log/messages ... snip ... May 9 12:23:46 ITS-ELST-01 logstash[1597324]: [2023-05-09T12:23:46,424][WARN][logstash.outputs.elasticsearch][main][d990eee626c9a5b7ee766183d9ecad6781d64a3cc7bd99486634821aa1cb3bb4] Could not index event to Elasticsearch. status: 400, ... snip ...l, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [1] shards, but this cluster currently has [4000]/[4000] maximum normal shards open;"}}} ``` I thought I was in an environment with no more than 4000 shards, but it turns out that the index life cycle is apparently not working. Also, the result of its\_index-policy-alias is as follows. ```auto # curl -XGET '/its_index-policy-alias?pretty' { "error" : { "root_cause" : [ { "type" : "index_not_found_exception", "reason" : "no such index [its_index-policy-alias]", "resource.type" : "index_or_alias", "resource.id" : "its_index-policy-alias", "index_uuid" : "_na_", "index" : "its_index-policy-alias" } ], "type" : "index_not_found_exception", "reason" : "no such index [its_index-policy-alias]", "resource.type" : "index_or_alias", "resource.id" : "its_index-policy-alias", "index_uuid" : "_na_", "index" : "its_index-policy-alias" }, "status" : 404 } ``` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [May 9, 2023, 4:01am UTC](https://discuss.elastic.co/t/my-lifecycle-policy-is-not-working/332856/4 "2023-05-09T04:01:48Z")
So you have a couple problems.. ILM works with a rollover alias You have no valid rollover alias Clients like logstash the elasticsearch output should be writing to the the rollover alias not the concret index Perhaps take a look at > **[Rollover | Elasticsearch Guide \[8.7\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-rollover.html)** Also share your logstash conf particularly the output section.. In short you are not writing to a rollover alias therefore the index can not rollover therefore ILM will not work. ---
**Author:** ![its-ogawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/its-ogawa/32/120829_2.png) [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)\ **Post date:** [May 9, 2023, 4:09am UTC](https://discuss.elastic.co/t/my-lifecycle-policy-is-not-working/332856/5 "2023-05-09T04:09:29Z")
The logstash conf is as follows: ```auto # The # character at the beginning of a line indicates a comment. # Use comments to describe your configuration. input { beats { port => "5044" } } # The filter part of this file is commented out to indicate that # it is optional. filter { ... snip ... } output { if([fields][index_name]) { elasticsearch { hosts => ["localhost"] index => "%{[fields][index_name]}-%{+YYYY.MM.dd}" } } else { elasticsearch { hosts => ["localhost"] index => "other-%{+YYYY.MM.dd}" } } } ``` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [May 9, 2023, 4:32am UTC](https://discuss.elastic.co/t/my-lifecycle-policy-is-not-working/332856/6 "2023-05-09T04:32:46Z")
You need to read the conceptd in the doc > **[Tutorial: Automate rollover with ILM | Elasticsearch Guide \[8.7\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html#getting-started-index-lifecycle-management)** You need to setup an initial managed index with the write rollover alias > **[Tutorial: Automate rollover with ILM | Elasticsearch Guide \[8.7\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html#ilm-gs-alias-bootstrap)** ```auto # Create Initial Managed index PUT my-index-2023.05.06-000001 { "aliases": { "its_index-policy-alias":{ "is_write_index": true } } } ``` Then you need to point logstash to write to the write alias. This only does daily index not ILM with rollover which will not work ```auto index => "other-%{+YYYY.MM.dd}" ``` Should be writing to the alias ```auto index => "its_index-policy-alias" ``` ---
**Author:** ![its-ogawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/its-ogawa/32/120829_2.png) [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)\ **Post date:** [May 9, 2023, 5:50am UTC](https://discuss.elastic.co/t/my-lifecycle-policy-is-not-working/332856/7 "2023-05-09T05:50:38Z")
Thank you for the information you have provided. I am still trying to understand the rollover alias. I maintain a number of logs and have defined an index\_name in filebeat for each. If I want to use that index\_name, how do I do it? In the example above, I would write ```auto index => "%{[fields][index_name]}-%{+YYYYY.MM.dd}" ``` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [May 9, 2023, 6:04am UTC](https://discuss.elastic.co/t/my-lifecycle-policy-is-not-working/332856/8 "2023-05-09T06:04:34Z")
When you use rollover alias is does not include the date... Read the docs I actually gave you the solution... Perhaps look at this > [@Filebeat process different log paths and write data to seperate index,Without use of logstash and follow ILM/rollover alias defined in template](https://discuss.elastic.co/t/filebeat-process-different-log-paths-and-write-data-to-seperate-index-without-use-of-logstash-and-follow-ilm-rollover-alias-defined-in-template/332593/12): > > @PRASHANT_MEHTA Looks at this carefully this is minimal but does everything you ask. This is the order you need to do this as well ... BTW timeseries indices are of the pattern my-index-yyyy.mm.dd-000001 # Create Common ILM Policy PUT \_ilm/policy/mis-monitoring-common-policy { "policy": { "phases": { "hot": { "actions": { "rollover": { "max\_age": "1d", "max\_size": "50gb" }, "set\_priority": { "priority": 10… ---
**Author:** ![its-ogawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/its-ogawa/32/120829_2.png) [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)\ **Post date:** [May 9, 2023, 6:36am UTC](https://discuss.elastic.co/t/my-lifecycle-policy-is-not-working/332856/9 "2023-05-09T06:36:09Z")
> [@stephenb](#): > > When you use rollover alias is does not include the date... That surprises me! Even without the `{+YYYY.MM.dd}`, is it still managed by date in the index? * * * I actually tried to do this. I can't see the index date... ```auto # curl -XGET http://localhost:9200/_cat/indices?v health index uuid pri rep docs.count docs.deleted store.size pri.store.size green open example ... snip ... ``` Does this mean that it will keep logging for one index for a given lifecycle? * * * How do I create rollover\_alias? > [@stephenb](#): > > ```auto > # Create Initial Managed index > PUT my-index-2023.05.06-000001 > { > "aliases": { > "its_index-policy-alias":{ > "is_write_index": true > } > } > } > > ``` Also, I already manage nearly 4000 indexes. Do I need to write a statement to apply rollover\_alias to all indexes? ---
**Author:** ![its-ogawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/its-ogawa/32/120829_2.png) [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)\ **Post date:** [May 11, 2023, 1:56am UTC](https://discuss.elastic.co/t/my-lifecycle-policy-is-not-working/332856/11 "2023-05-11T01:56:58Z")
I created several new policies and worked on understanding ILM. Stephen eagerly tried to show me how to solve it, but I finally couldn't master it. Instead, I enlisted the help of Curator. This worked out well, and I am very happy with the results. > **[delete\_indices | Curator Reference \[8.0\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/client/curator/current/ex_delete_indices.html)** I hope this helps others who are having the same problem. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [May 11, 2023, 2:09am UTC](https://discuss.elastic.co/t/my-lifecycle-policy-is-not-working/332856/12 "2023-05-11T02:09:47Z")
> [@its-ogawa](#): > > Also, I already manage nearly 4000 indexes. You need to clean up indices to make room > [@its-ogawa](#): > > Do I need to write a statement to apply rollover\_alias to all indexes? Apologies, but no all those indices are too late for rollover I was trying to give you the solution for "Go Forward" If you wanted to manage existing then you could have looked at this Manually Create a Policy > **[Manage existing indices | Elasticsearch Guide \[8.11\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-with-existing-indices.html#ilm-existing-indices-apply)** And Manually Applied to every Index > **[Index lifecycle management settings in Elasticsearch | Elasticsearch Guide...](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-settings.html#_index_level_settings)** Which then would parse the date of the index and apply the ILM Policy ... that would not work for new indices only old.. This may still work for you.... > [@its-ogawa](#): > > Instead, I enlisted the help of Curator. This worked out well, and I am very happy with the results. Good! I think you could have done the same with the manual ILM but glad you got it working... ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 8, 2023, 2:10am UTC](https://discuss.elastic.co/t/my-lifecycle-policy-is-not-working/332856/13 "2023-06-08T02:10:12Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.