# Index Lifecycle Management rolleover not happening properly **URL:** \ **Category:** Kibana\ **Tags:** elastic-stack-monitoring\ **Created:** [October 12, 2020, 1:35pm UTC](https://discuss.elastic.co/t/index-lifecycle-management-rolleover-not-happening-properly/251771 "2020-10-12T13:35:35Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![amiya-santra](https://avatars.discourse-cdn.com/v4/letter/a/bc8723/32.png) [@amiya-santra](https://discuss.elastic.co/u/amiya-santra)\ **Post date:** [October 12, 2020, 1:35pm UTC](https://discuss.elastic.co/t/index-lifecycle-management-rolleover-not-happening-properly/251771/1 "2020-10-12T13:35:36Z")
I have manually configured ILM. After creating policies and telling my indices to use them, they all complain with errors like this: ```auto issue: rollover index have 0 doc ``` ```auto PUT _ilm/policy/new_policy { "policy": { "phases": { "hot": { "min_age": "0ms", "actions": { "rollover": { "max_age": "2h", "max_size": "100mb" }, "set_priority": { "priority": 10 } } }, "delete": { "min_age": "1h", "actions": { "delete": {} } } } } } ``` ```auto Template PUT _template/prod-log-temp { "index_patterns": [ "my-log-*" ], "settings": { "index": { "lifecycle": { "name": "new_policy", "rollover_alias": "my-log" } } } } ``` My policy doing rollover index but its containing 0 doc first index: my-log-00001(only incrising size of this index) second index: my-log-00002 (having 0 doc) Can any one help why this is happening. ---
**Author:** ![kavierkoo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kavierkoo/32/86555_2.png) [@kavierkoo](https://discuss.elastic.co/u/kavierkoo)\ **Post date:** [October 12, 2020, 2:35pm UTC](https://discuss.elastic.co/t/index-lifecycle-management-rolleover-not-happening-properly/251771/2 "2020-10-12T14:35:27Z")
> [@amiya-santra](#): > > first index: my-log-00001(only incrising size of this index) > second index: my-log-00002 (having 0 doc) I'm suspecting your logstash conf's output for this specific index is not pointing to ILM alias, Could you share the logstash .conf's output? ---
**Author:** ![amiya-santra](https://avatars.discourse-cdn.com/v4/letter/a/bc8723/32.png) [@amiya-santra](https://discuss.elastic.co/u/amiya-santra)\ **Post date:** [October 13, 2020, 2:34am UTC](https://discuss.elastic.co/t/index-lifecycle-management-rolleover-not-happening-properly/251771/3 "2020-10-13T02:34:22Z")
Thanks for your quick response. I am not using logstash for transferring log I am using filebeat, here is my filebeat output configuration. ```auto output.elasticsearch: hosts: ['${ELASTICSEARCH_HOST:13.xx.xx.xx}:${ELASTICSEARCH_PORT:31116}'] username: ${ELASTICSEARCH_USERNAME} password: ${ELASTICSEARCH_PASSWORD} indices: - index: "%{[type]}-000001" when.equals: type: "my-log" setup.ilm.enabled: false ilm.enabled: false ``` ---
**Author:** ![kavierkoo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kavierkoo/32/86555_2.png) [@kavierkoo](https://discuss.elastic.co/u/kavierkoo)\ **Post date:** [October 13, 2020, 6:32am UTC](https://discuss.elastic.co/t/index-lifecycle-management-rolleover-not-happening-properly/251771/4 "2020-10-13T06:32:33Z")
> [@amiya-santra](#): > > `index: "%{[type]}-000001"` As I was suspected, your `index` is pointing to `%{[type]}-000001` hence the ilm not working. You need to point to your `alias` instead of an index name with the 000001. Some useful info: > **[Tutorial: Automate rollover with ILM | Elasticsearch Guide \[8.11\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html#manage-time-series-data-without-data-streams)** Hope this could help! ---
**Author:** ![amiya-santra](https://avatars.discourse-cdn.com/v4/letter/a/bc8723/32.png) [@amiya-santra](https://discuss.elastic.co/u/amiya-santra)\ **Post date:** [October 13, 2020, 8:47am UTC](https://discuss.elastic.co/t/index-lifecycle-management-rolleover-not-happening-properly/251771/5 "2020-10-13T08:47:58Z")
But my `%{[type]}` parameter value is ``` my-log ```auto It means both are the same [type] value and my alias. so you mean to say don't need to provide index as index: "%{[type]}-000001" I can write like this index: "%{[type]}". ``` ---
**Author:** ![kavierkoo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kavierkoo/32/86555_2.png) [@kavierkoo](https://discuss.elastic.co/u/kavierkoo)\ **Post date:** [October 13, 2020, 9:06am UTC](https://discuss.elastic.co/t/index-lifecycle-management-rolleover-not-happening-properly/251771/6 "2020-10-13T09:06:46Z")
If your `%{[type]}` match your `rollover-alias` it should work. But before you do the changes, do you mind to share below info for a better understanding? - `GET /my-log-00001` (Just wanted to see the aliases portion) - `GET /my-log-00002` (Just wanted to see the aliases portion) - `GET my-log-00001/_ilm/ explain` - `GET my-log-00002/_ilm/ explain` ---
**Author:** ![amiya-santra](https://avatars.discourse-cdn.com/v4/letter/a/bc8723/32.png) [@amiya-santra](https://discuss.elastic.co/u/amiya-santra)\ **Post date:** [October 14, 2020, 5:54am UTC](https://discuss.elastic.co/t/index-lifecycle-management-rolleover-not-happening-properly/251771/7 "2020-10-14T05:54:27Z")
> [@kavierkoo](#): > > GET my-log-00001/\_ilm/ explain Now its creating 2 indexes after that its giving this error illegal\_argument\_exception: index.lifecycle.rollover\_alias [prod-nmn] does not point to index [prod-nmn-000001] GET prod-nmn-000001/\_ilm/ explain ```auto { "indices" : { "prod-nmn-000001" : { "index" : "prod-nmn-000001", "managed" : true, "policy" : "new_policy", "lifecycle_date_millis" : 1602650088714, "age" : "28.23m", "phase" : "hot", "phase_time_millis" : 1602650088802, "action" : "rollover", "action_time_millis" : 1602650621674, "step" : "ERROR", "step_time_millis" : 1602651221583, "failed_step" : "check-rollover-ready", "step_info" : { "type" : "illegal_argument_exception", "reason" : "index.lifecycle.rollover_alias [prod-nmn] does not point to index [prod-nmn-000001]", "stack_trace" : """ java.lang.IllegalArgumentException: index.lifecycle.rollover_alias [prod-nmn] does not point to index [prod-nmn-000001] at org.elasticsearch.xpack.core.ilm.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:92) at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:133) at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:274) at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggered(IndexLifecycleService.java:213) at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:175) at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:203) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:830) """ }, "phase_execution" : { "policy" : "new_policy", "phase_definition" : { "min_age" : "0ms", "actions" : { "rollover" : { "max_size" : "100mb", "max_age" : "2h" }, "set_priority" : { "priority" : 10 } } }, "version" : 3, "modified_date_in_millis" : 1602499425529 } } } } ``` `GET /prod-nmn-000001` ```auto { "prod-nmn-000001" : { "aliases" : { }, "mappings" : { "properties" : { "@timestamp" : { "type" : "date" }, "agent" : { "properties" : { "ephemeral_id" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "hostname" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "id" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "type" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "version" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } } } }, "ecs" : { "properties" : { "version" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } } } }, "host" : { "properties" : { "name" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } } } }, "input" : { "properties" : { "type" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } } } }, "log" : { "properties" : { "file" : { "properties" : { "path" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } } } }, "flags" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "gw_timestamp" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "loglevel" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "messages" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "network-id" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "offset" : { "type" : "long" } } }, "message" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "type" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } } } }, "settings" : { "index" : { "lifecycle" : { "name" : "new_policy", "rollover_alias" : "prod-nmn" }, "number_of_shards" : "1", "provided_name" : "prod-nmn-000001", "creation_date" : "1602650088714", "priority" : "10", "number_of_replicas" : "0", "uuid" : "-E-yDN6hQFmRxpfl_gia0w", "version" : { "created" : "7040299" } } } } } ``` ---
**Author:** ![kavierkoo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kavierkoo/32/86555_2.png) [@kavierkoo](https://discuss.elastic.co/u/kavierkoo)\ **Post date:** [October 14, 2020, 6:31am UTC](https://discuss.elastic.co/t/index-lifecycle-management-rolleover-not-happening-properly/251771/8 "2020-10-14T06:31:13Z")
> [@amiya-santra](#): > > `index.lifecycle.rollover_alias [prod-nmn] does not point to index [prod-nmn-000001]` > [@amiya-santra](#): > > ```auto > { > "prod-nmn-000001" : { > "aliases" : { }, > > ``` It shows alias is empty for this index. Did you bootstrap your index after you created your policy and template? ILM require below steps: 1. Create a lifecycle policy that defines the appropriate phases and actions. See [Create a lifecycle policy](https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html#ilm-gs-create-policy) above. 2. [Create an index template](https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html#ilm-gs-alias-apply-policy) to apply the policy to each new index. 3. [Bootstrap an index](https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html#ilm-gs-alias-bootstrap) as the initial write index. 4. [Verify indices are moving through the lifecycle phases](https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html#ilm-gs-alias-check-progress) as expected. > **[Tutorial: Automate rollover with ILM | Elasticsearch Guide \[8.11\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html#ilm-gs-alias-bootstrap)** also, isn't the index name `my-log-xxxxx`, why the log shows `prod-nmn-xxxxxx` instead? ---
**Author:** ![amiya-santra](https://avatars.discourse-cdn.com/v4/letter/a/bc8723/32.png) [@amiya-santra](https://discuss.elastic.co/u/amiya-santra)\ **Post date:** [October 14, 2020, 7:00am UTC](https://discuss.elastic.co/t/index-lifecycle-management-rolleover-not-happening-properly/251771/9 "2020-10-14T07:00:10Z")
Thanks Kavier, I changed the index name from my-log to prod-nmn. can we do it dynamically? because my index is going to create dynamically so how to do bootstrap for dynamically created index? as I check in example ```auto PUT timeseries-000001 { "aliases": { "timeseries": { "is_write_index": true } } } ``` they are bootstrapping only for this index timeseries-000001 what about other index which ae going to create after rollover? ---
**Author:** ![kavierkoo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kavierkoo/32/86555_2.png) [@kavierkoo](https://discuss.elastic.co/u/kavierkoo)\ **Post date:** [October 14, 2020, 7:16am UTC](https://discuss.elastic.co/t/index-lifecycle-management-rolleover-not-happening-properly/251771/10 "2020-10-14T07:16:12Z")
> [@amiya-santra](#): > > can we do it dynamically? because my index is going to create dynamically so how to do bootstrap for dynamically created index This is where the `index template` come in as you configured below. > [@amiya-santra](#): > > ```auto > "settings": { > "index": { > "lifecycle": { > "name": "new_policy", > "rollover_alias": "my-log" > } > } > > ``` you only have to bootstap the first one, and subsequently as long as the `index pattern` in your `index template` matched, it will auto assign the `rollover alias` to your `xxx-000002` index > [@amiya-santra](#): > > ```auto > "index_patterns": [ > "my-log-*" > ], > > ``` I strongly suggest to have a read up on the whole [ILM document](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html) as it is really helpful to understand how ILM works. Also I find [Datasteam](https://www.elastic.co/guide/en/elasticsearch/reference/master/data-streams.html) doc helps me understand ILM even better. Hope this can help you! ---
**Author:** ![amiya-santra](https://avatars.discourse-cdn.com/v4/letter/a/bc8723/32.png) [@amiya-santra](https://discuss.elastic.co/u/amiya-santra)\ **Post date:** [October 14, 2020, 11:19am UTC](https://discuss.elastic.co/t/index-lifecycle-management-rolleover-not-happening-properly/251771/11 "2020-10-14T11:19:58Z")
Thanks Kavier, I went through the document it's help a lot to understand ILM . one small question I am giving rollover index size 100mb but my rollover index are containing more then 100 mb log like 350Mb,284Mb,456Mb. Why it's happening like this? ---
**Author:** ![kavierkoo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kavierkoo/32/86555_2.png) [@kavierkoo](https://discuss.elastic.co/u/kavierkoo)\ **Post date:** [October 16, 2020, 5:18am UTC](https://discuss.elastic.co/t/index-lifecycle-management-rolleover-not-happening-properly/251771/12 "2020-10-16T05:18:09Z")
Can you run another round of below commands to have a better idea: - `GET INDEXNAME-000001/_ilm/ explain` - [Dry run](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-rollover-index.html#rollover-index-api-dry-run-ex) following your existing policy for that specific index ---
**Author:** ![amiya-santra](https://avatars.discourse-cdn.com/v4/letter/a/bc8723/32.png) [@amiya-santra](https://discuss.elastic.co/u/amiya-santra)\ **Post date:** [October 16, 2020, 7:44am UTC](https://discuss.elastic.co/t/index-lifecycle-management-rolleover-not-happening-properly/251771/13 "2020-10-16T07:44:01Z")
I applied the policy and template in a new alias "0000000002000001000049cd-5d1352be83a3f2001718fba5-5dcfe8942a455a0016be0296" GET 0000000002000001000049cd-5d1352be83a3f2001718fba5-5dcfe8942a455a0016be0296-000001/\_ilm/explain ```auto { "indices" : { "0000000002000001000049cd-5d1352be83a3f2001718fba5-5dcfe8942a455a0016be0296-000001" : { "index" : "0000000002000001000049cd-5d1352be83a3f2001718fba5-5dcfe8942a455a0016be0296-000001", "managed" : true, "policy" : "new-device-policy", "lifecycle_date_millis" : 1602825221945, "age" : "2.38h", "phase" : "hot", "phase_time_millis" : 1602824303020, "action" : "complete", "action_time_millis" : 1602825222393, "step" : "complete", "step_time_millis" : 1602825222393, "phase_execution" : { "policy" : "new-device-policy", "phase_definition" : { "min_age" : "0ms", "actions" : { "rollover" : { "max_size" : "1mb", "max_age" : "1h" } } }, "version" : 1, "modified_date_in_millis" : 1602819713971 } } } } ``` I did a dry run on my alias with max age 1 day and max size 1 mb as i mention in policy. ```auto POST /0000000002000001000049cd-5d1352be83a3f2001718fba5-5dcfe8942a455a0016be0296/_rollover?dry_run { "conditions" : { "max_age": "1d", "max_size": "1mb" } } ``` Response ```auto { "acknowledged" : false, "shards_acknowledged" : false, "old_index" : "0000000002000001000049cd-5d1352be83a3f2001718fba5-5dcfe8942a455a0016be0296-000009", "new_index" : "0000000002000001000049cd-5d1352be83a3f2001718fba5-5dcfe8942a455a0016be0296-000010", "rolled_over" : false, "dry_run" : true, "conditions" : { "[max_age: 1d]" : false, "[max_size: 1mb]" : true } } ``` where i am doing wrong? ---
**Author:** ![kavierkoo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kavierkoo/32/86555_2.png) [@kavierkoo](https://discuss.elastic.co/u/kavierkoo)\ **Post date:** [October 16, 2020, 7:59am UTC](https://discuss.elastic.co/t/index-lifecycle-management-rolleover-not-happening-properly/251771/14 "2020-10-16T07:59:25Z")
> [@amiya-santra](#): > > GET 0000000002000001000049cd-5d1352be83a3f2001718fba5-5dcfe8942a455a0016be0296-000001/\_ilm/explain After this one did you see the index got rollover to xxx-000002? ---
**Author:** ![amiya-santra](https://avatars.discourse-cdn.com/v4/letter/a/bc8723/32.png) [@amiya-santra](https://discuss.elastic.co/u/amiya-santra)\ **Post date:** [October 16, 2020, 8:09am UTC](https://discuss.elastic.co/t/index-lifecycle-management-rolleover-not-happening-properly/251771/15 "2020-10-16T08:09:08Z")
index are getting rollover by its own, now I have total 10 index after getting rollover and it keeps going. after doing a dry run in responce its printing old index and new index but logs are going to the old index, no new index is getting created immediately its happening by its own after some time. ```auto { "acknowledged" : false, "shards_acknowledged" : false, "old_index" : "0000000002000001000049cd-5d1352be83a3f2001718fba5-5dcfe8942a455a0016be0296-000010", "new_index" : "0000000002000001000049cd-5d1352be83a3f2001718fba5-5dcfe8942a455a0016be0296-000011", "rolled_over" : false, "dry_run" : true, "conditions" : { "[max_age: 1d]" : false, "[max_size: 1mb]" : true } } ``` ---
**Author:** ![kavierkoo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kavierkoo/32/86555_2.png) [@kavierkoo](https://discuss.elastic.co/u/kavierkoo)\ **Post date:** [October 16, 2020, 8:32am UTC](https://discuss.elastic.co/t/index-lifecycle-management-rolleover-not-happening-properly/251771/16 "2020-10-16T08:32:38Z")
Dry run is a simulation, it should not execute the rollover action. Anyway, it seems like its working already isnt it? > [@amiya-santra](#): > > after doing a dry run in responce its printing old index and new index but logs are going to the old index, I dont really get this part, could you elaborate on this? ---
**Author:** ![amiya-santra](https://avatars.discourse-cdn.com/v4/letter/a/bc8723/32.png) [@amiya-santra](https://discuss.elastic.co/u/amiya-santra)\ **Post date:** [October 16, 2020, 8:39am UTC](https://discuss.elastic.co/t/index-lifecycle-management-rolleover-not-happening-properly/251771/17 "2020-10-16T08:39:42Z")
if it's doing a dry run then ``` acknowledged ```auto yes, it's working but the size I have provided in policy to rollover (2mb) so rollover index should container 2 mb log not more then that .. but my rollover index contain 25 to 60 mb logs.. why its happenig? ``` ---
**Author:** ![kavierkoo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kavierkoo/32/86555_2.png) [@kavierkoo](https://discuss.elastic.co/u/kavierkoo)\ **Post date:** [October 16, 2020, 9:00am UTC](https://discuss.elastic.co/t/index-lifecycle-management-rolleover-not-happening-properly/251771/18 "2020-10-16T09:00:38Z")
There's an interval of 10 minutes that rollover will happen > `indices.lifecycle.poll_interval` > ([Dynamic](https://www.elastic.co/guide/en/elasticsearch/reference/current/settings.html#dynamic-cluster-setting), [time unit value](https://www.elastic.co/guide/en/elasticsearch/reference/current/common-options.html#time-units)) How often index lifecycle management checks for indices that meet policy criteria. Defaults to `10m` . > **[Index lifecycle management settings in Elasticsearch | Elasticsearch Guide...](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-settings.html)** Rollover will only run after every interval, if within the interval your logs keeps coming in, it will still store at the previous even after the policy hits. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 13, 2020, 9:00am UTC](https://discuss.elastic.co/t/index-lifecycle-management-rolleover-not-happening-properly/251771/19 "2020-11-13T09:00:47Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.