# No rollover with data stream lifecycle
**URL:** https://discuss.elastic.co/t/no-rollover-with-data-stream-lifecycle/382165
**Category:** Elasticsearch
**Created:** [September 23, 2025, 1:04pm UTC](https://discuss.elastic.co/t/no-rollover-with-data-stream-lifecycle/382165 "2025-09-23T13:04:16Z")
**Posts on this page:** 13
**Page:** 1
### Author:  [@mikehaertl](https://discuss.elastic.co/u/mikehaertl)
#### Post date: [September 23, 2025, 1:04pm UTC](https://discuss.elastic.co/t/no-rollover-with-data-stream-lifecycle/382165/1 "2025-09-23T13:04:16Z")
For testing purposes I’ve configured a lifecycle on a data stream with a retention time of `1h`. But the backing index does not get rolled over.
According to this doc [Data stream lifecycle settings in Elasticsearch | Reference](https://www.elastic.co/docs/reference/elasticsearch/configuration-reference/data-stream-lifecycle-settings#cluster-lifecycle-default-rollover) with the default setting of `cluster.lifecycle.default.rollover` the rollover should happen if:
> - Either any primary shard reaches the size of 50GB,
>
> - or any primary shard contains 200.000.000 documents
>
> - or the index reaches a certain age _ **which depends on the retention time of your data stream** _,
>
> - **and** has at least one document.
So in my case I would expect a rollover after `1h`.
If I explain the lifecycle on the data stream I see that the lifecycle is properly configured on the backing index:
```auto
GET logs-prod2-nlc/_lifecycle/explain
{
"indices": {
".ds-logs-prod2-nlc-2025.09.22-000001": {
"index": ".ds-logs-prod2-nlc-2025.09.22-000001",
"managed_by_lifecycle": true,
"index_creation_date_millis": 1758554925361,
"time_since_index_creation": "21.35h",
"lifecycle": {
"enabled": true,
"data_retention": "1h",
"effective_retention": "1h",
"retention_determined_by": "data_stream_configuration"
}
}
}
}
```
I also checked that `prefer_ilm` is set to `false` in the backing index settings:
```auto
{
"settings": {
"index": {
...
"number_of_replicas": "0",
"uuid": "faU5gfEnQgmZsukYd6O-AA",
"version": {
"created": "9033000"
},
"lifecycle": {
"name": "logs",
"prefer_ilm": "false"
},
```
So why is there no rollover? What am I missing?
---
### Author:  [@Keith\_Massey](https://discuss.elastic.co/u/Keith_Massey)
#### Post date: [September 23, 2025, 3:22pm UTC](https://discuss.elastic.co/t/no-rollover-with-data-stream-lifecycle/382165/2 "2025-09-23T15:22:58Z")
Are there any errors in the logs of any of your elasticsearch nodes? Maybe rollover is failing for some reason? If you don’t see any errors in any logs, I would check the health of the cluster (`GET _health_report`). And if the reported health is green, I would probably turn on trace-level logging for `org.elasticsearch.datastreams.lifecycle.DataStreamLifecycleService` and `org.elasticsearch.action.admin.indices.rollover.TransportRolloverAction` (this will be verbose, so I don’t recommend it in production, at least not for very long).
For what it’s worth, I just tried this out by copy/pasting the commands in the [data stream lifecycle tutorial](https://www.elastic.co/docs/manage-data/lifecycle/data-stream/tutorial-create-data-stream-with-lifecycle), and it worked as you would expect. The only change I made was that I reduced the retention from 7d to 1h. Shortly after the 1h mark, my write index rolled over.
---
### Author:  [@mikehaertl](https://discuss.elastic.co/u/mikehaertl)
#### Post date: [September 23, 2025, 4:18pm UTC](https://discuss.elastic.co/t/no-rollover-with-data-stream-lifecycle/382165/3 "2025-09-23T16:18:42Z")
Thanks, I will check the TRACE log.
Meanwhile it rolled over - but probably for other reasons that I’d expected:
> Desired balance computation for [37] is still not converged after [3.2h] and [1] iterations
> Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.ds-logs-prod2-nlc-2025. 09.23-000002][0]]]).
> Data stream lifecycle successfully rolled over datastream [logs-prod2-nlc] due to the following met _**rollover conditions [[max\_age: 1 d]**_, [min\_docs: 1]]. The new index is [.ds-logs-prod2-nlc-2025.09.23-000002]
> [.ds-logs-prod2-nlc-2025.09.23-000002/gjKaOODPRvGWUCdaH8qjVQ] update\_mapping [\_doc]
> [.ds-logs-prod2-nlc-2025.09.23-000002/gjKaOODPRvGWUCdaH8qjVQ] update\_mapping [\_doc]
> [.ds-logs-prod2-nlc-2025.09.23-000002/gjKaOODPRvGWUCdaH8qjVQ] update\_mapping [\_doc]
> Data stream lifecycle service successfully updated settings [[index.merge.policy.merge\_factor, index.merge.policy.floor\_segment]] for index index [.ds-logs-prod2-nlc-2025.09.22-000001]
> Data stream lifecycle is issuing a request to _ **force merge** _ index [.ds-logs-prod2-nlc-2025.09.22-000001]
> Data stream lifecycle successfully force merged index [.ds-logs-prod2-nlc-2025.09.22-000001]
> [.ds-logs-prod2-nlc-2025.09.23-000002/gjKaOODPRvGWUCdaH8qjVQ] update\_mapping [\_doc]
It looks like the rollover happened after `1d`.
Also is it normal to see a force merge? The docs say that it should rather do a “tail merge” (whatever that exactly means)…
Could it be that I still see some remains of the previously configured ILM policy on the index (even though `prefer_ilm` is `false` now)?
---
### Author:  [@Keith\_Massey](https://discuss.elastic.co/u/Keith_Massey)
#### Post date: [September 23, 2025, 5:58pm UTC](https://discuss.elastic.co/t/no-rollover-with-data-stream-lifecycle/382165/4 "2025-09-23T17:58:11Z")
> [@mikehaertl](#):
>
> Also is it normal to see a force merge? The docs say that it should rather do a “tail merge” (whatever that exactly means)…
Yes that is normal. The tail merge that data stream lifecycles do is just a force merge that is optimized for runtime performance – it merges segments but not down to a single segment.
---
### Author:  [@Keith\_Massey](https://discuss.elastic.co/u/Keith_Massey)
#### Post date: [September 23, 2025, 6:02pm UTC](https://discuss.elastic.co/t/no-rollover-with-data-stream-lifecycle/382165/5 "2025-09-23T18:02:53Z")
> [@mikehaertl](#):
>
> Could it be that I still see some remains of the previously configured ILM policy on the index (even though `prefer_ilm` is `false` now)?
I don’t think so – everything in that log looks data-stream-lifecycle-related. Is there any chance you changed your retention after the data stream was created, and elasticsearch did not correctly update the rollover to reflect that change? Here are the rollover rules in a comment in the code: [elasticsearch/server/src/main/java/org/elasticsearch/action/admin/indices/rollover/RolloverConfiguration.java at main · elastic/elasticsearch · GitHub](https://github.com/elastic/elasticsearch/blob/main/server/src/main/java/org/elasticsearch/action/admin/indices/rollover/RolloverConfiguration.java#L174-L179) .
---
### Author:  [@mikehaertl](https://discuss.elastic.co/u/mikehaertl)
#### Post date: [September 25, 2025, 6:38am UTC](https://discuss.elastic.co/t/no-rollover-with-data-stream-lifecycle/382165/6 "2025-09-25T06:38:41Z")
> [@Keith\_Massey](#):
>
> Is there any chance you changed your retention after the data stream was created, and elasticsearch did not correctly update the rollover to reflect that change?
Yes, that’s very well possible. Could be that I started with `1d` and then changed my mind and set it to `1h`. The rollover still happens after `1d` now.
So would you consider this a bug? Anything I should try to boil it down further?
Probably not related but I also noticed something curious: The index suffix went from `000002` to `000004` and skipped `000003`:
> "Data stream lifecycle is issuing a request to force merge index [.ds-logs-prod2-nlc-2025.09.23-000002]"
> "Data stream lifecycle successfully force merged index [.ds-logs-prod2-nlc-2025.09.23-000002]"
> "Updating cluster state with force merge complete marker for .ds-logs-prod2-nlc-2025.09.23-000002"
> "Updated cluster state for force merge of index [.ds-logs-prod2-nlc-2025.09.23-000002]"
> "Clearing recorded error for index [.ds-logs-prod2-nlc-2025.09.23-000002] because the [indices:admin/forcemerge] action was successful"
> "Data stream lifecycle issues rollover request for data stream [logs-prod2-nlc]"
> "Already force merged .ds-logs-prod2-nlc-2025.09.23-000002"
> "auto sharding result for data stream [logs-prod2-nlc] is [No recommendation as auto-sharding not enabled]"
> "Clearing recorded error for index [.ds-logs-prod2-nlc-2025.09.24-000004] because the [indices:admin/rollover] action was successful"
> "Data stream lifecycle issues rollover request for data stream [logs-prod2-nlc]"
> "Already force merged .ds-logs-prod2-nlc-2025.09.23-000002"
> "auto sharding result for data stream [logs-prod2-nlc] is [No recommendation as auto-sharding not enabled]"
In the data stream it also claims that the generation is `5` now. But I only count 3 rollovers so far.
---
### Author:  [@Keith\_Massey](https://discuss.elastic.co/u/Keith_Massey)
#### Post date: [September 25, 2025, 1:23pm UTC](https://discuss.elastic.co/t/no-rollover-with-data-stream-lifecycle/382165/7 "2025-09-25T13:23:24Z")
> [@mikehaertl](#):
>
> So would you consider this a bug?
Yeah that would be a bug. Can you reproduce it? If I have time, I’ll try to reproduce it today.
> [@mikehaertl](#):
>
> Probably not related but I also noticed something curious: The index suffix went from `000002` to `000004` and skipped `000003`
That’s perfectly normal. You’ll see that pretty frequently. I forget exactly why that is the case, but there is no guarantee that those numbers will not have gaps.
---
### Author:  [@mikehaertl](https://discuss.elastic.co/u/mikehaertl)
#### Post date: [September 25, 2025, 2:09pm UTC](https://discuss.elastic.co/t/no-rollover-with-data-stream-lifecycle/382165/8 "2025-09-25T14:09:21Z")
> [@Keith\_Massey](#):
>
> Can you reproduce it?
I think so. Here’s what I tried today:
1. Create a `logs@custom` template component (to be included by the default `logs` template)
2. Set a data retention of `2h` there and also `index.lifecylce.prefer_ihm` to `false`
3. Delete an already created data stream
4. Wait for the data stream to be recreated (it constantly receives logs via logstash)
Now if I check the data stream all seems to look fine: data retention is `2h`.
But there’s no rollover after 2h.
> **Preview of the “logs” template**
>
> ```json
> {
> "template": {
> "settings": {
> "index": {
> "lifecycle": {
> "name": "logs",
> "prefer_ilm": "false"
> },
> "codec": "best_compression",
> "routing": {
> "allocation": {
> "include": {
> "_tier_preference": "data_hot"
> }
> }
> },
> "default_pipeline": "logs@default-pipeline",
> "mapping": {
> "total_fields": {
> "ignore_dynamic_beyond_limit": "true"
> },
> "ignore_malformed": "true"
> },
> "number_of_replicas": "0"
> }
> },
> "mappings": {
> "_data_stream_timestamp": {
> "enabled": true
> },
> "dynamic_templates": [
> {
> "ecs_timestamp": {
> "match": "@timestamp",
> "mapping": {
> "ignore_malformed": false,
> "type": "date"
> }
> }
> },
> {
> "ecs_message_match_only_text": {
> "path_match": [
> "message",
> "*.message"
> ],
> "unmatch_mapping_type": "object",
> "mapping": {
> "type": "match_only_text"
> }
> }
> },
> {
> "ecs_non_indexed_keyword": {
> "path_match": [
> "*event.original",
> "*gen_ai.agent.description"
> ],
> "mapping": {
> "doc_values": false,
> "index": false,
> "type": "keyword"
> }
> }
> },
> {
> "ecs_non_indexed_long": {
> "path_match": "*.x509.public_key_exponent",
> "mapping": {
> "doc_values": false,
> "index": false,
> "type": "long"
> }
> }
> },
> {
> "ecs_ip": {
> "path_match": [
> "ip",
> "*.ip",
> "*_ip"
> ],
> "match_mapping_type": "string",
> "mapping": {
> "type": "ip"
> }
> }
> },
> {
> "ecs_wildcard": {
> "path_match": [
> "*.io.text",
> "*.message_id",
> "*registry.data.strings",
> "*url.path"
> ],
> "unmatch_mapping_type": "object",
> "mapping": {
> "type": "wildcard"
> }
> }
> },
> {
> "ecs_path_match_wildcard_and_match_only_text": {
> "path_match": [
> "*.body.content",
> "*url.full",
> "*url.original"
> ],
> "unmatch_mapping_type": "object",
> "mapping": {
> "fields": {
> "text": {
> "type": "match_only_text"
> }
> },
> "type": "wildcard"
> }
> }
> },
> {
> "ecs_match_wildcard_and_match_only_text": {
> "match": [
> "*command_line",
> "*stack_trace"
> ],
> "unmatch_mapping_type": "object",
> "mapping": {
> "fields": {
> "text": {
> "type": "match_only_text"
> }
> },
> "type": "wildcard"
> }
> }
> },
> {
> "ecs_path_match_keyword_and_match_only_text": {
> "path_match": [
> "*.title",
> "*.executable",
> "*.name",
> "*.working_directory",
> "*.full_name",
> "*file.path",
> "*file.target_path",
> "*os.full",
> "*email.subject",
> "*vulnerability.description",
> "*user_agent.original"
> ],
> "unmatch_mapping_type": "object",
> "mapping": {
> "fields": {
> "text": {
> "type": "match_only_text"
> }
> },
> "type": "keyword"
> }
> }
> },
> {
> "ecs_date": {
> "path_match": [
> "*.timestamp",
> "*_timestamp",
> "*.not_after",
> "*.not_before",
> "*.accessed",
> "created",
> "*.created",
> "*.installed",
> "*.creation_date",
> "*.ctime",
> "*.mtime",
> "ingested",
> "*.ingested",
> "*.start",
> "*.end",
> "*.indicator.first_seen",
> "*.indicator.last_seen",
> "*.indicator.modified_at",
> "*threat.enrichments.matched.occurred"
> ],
> "unmatch_mapping_type": "object",
> "mapping": {
> "type": "date"
> }
> }
> },
> {
> "ecs_path_match_float": {
> "path_match": [
> "*.score.*",
> "*_score*"
> ],
> "path_unmatch": "*.version",
> "unmatch_mapping_type": "object",
> "mapping": {
> "type": "float"
> }
> }
> },
> {
> "ecs_usage_double_scaled_float": {
> "path_match": "*.usage",
> "match_mapping_type": [
> "double",
> "long",
> "string"
> ],
> "mapping": {
> "scaling_factor": 1000,
> "type": "scaled_float"
> }
> }
> },
> {
> "ecs_geo_point": {
> "path_match": "*.geo.location",
> "mapping": {
> "type": "geo_point"
> }
> }
> },
> {
> "ecs_flattened": {
> "path_match": [
> "*structured_data",
> "*exports",
> "*imports"
> ],
> "match_mapping_type": "object",
> "mapping": {
> "type": "flattened"
> }
> }
> },
> {
> "ecs_gen_ai_integers": {
> "path_match": [
> "*gen_ai.request.max_tokens",
> "*gen_ai.usage.input_tokens",
> "*gen_ai.usage.output_tokens",
> "*gen_ai.request.choice.count",
> "*gen_ai.request.seed"
> ],
> "mapping": {
> "type": "integer"
> }
> }
> },
> {
> "ecs_gen_ai_doubles": {
> "path_match": [
> "*gen_ai.request.temperature",
> "*gen_ai.request.top_k",
> "*gen_ai.request.frequency_penalty",
> "*gen_ai.request.presence_penalty",
> "*gen_ai.request.top_p"
> ],
> "mapping": {
> "type": "double"
> }
> }
> },
> {
> "all_strings_to_keywords": {
> "match_mapping_type": "string",
> "mapping": {
> "ignore_above": 1024,
> "type": "keyword"
> }
> }
> }
> ],
> "date_detection": false,
> "properties": {
> "@timestamp": {
> "type": "date",
> "ignore_malformed": false
> },
> "data_stream": {
> "properties": {
> "dataset": {
> "type": "constant_keyword"
> },
> "namespace": {
> "type": "constant_keyword"
> },
> "type": {
> "type": "constant_keyword",
> "value": "logs"
> }
> }
> }
> }
> },
> "aliases": {},
> "lifecycle": {
> "enabled": true,
> "data_retention": "2h"
> },
> "data_stream_options": {
> "failure_store": {
> "enabled": true
> }
> }
> }
> }
>
> ```
---
### Author:  [@mikehaertl](https://discuss.elastic.co/u/mikehaertl)
#### Post date: [September 29, 2025, 6:25am UTC](https://discuss.elastic.co/t/no-rollover-with-data-stream-lifecycle/382165/9 "2025-09-29T06:25:38Z")
I’ve now tried again with a retention time of `2d`. In this case everything seems to work fine:
- Backing indices are rolled over after `1d`(matching the code comment you linked above)
- For each data stream there are indices for the last 2 days
- Older indices get removed
So seems like the issue is only if retention time is \<= `1d`.
EDIT: Sorry, I think I got confused. I configured data retention to `1d` not `2d`. So I guess the result is not what I would have expected. I will try now again, this time really with `2d`.
---
### Author:  [@Keith\_Massey](https://discuss.elastic.co/u/Keith_Massey)
#### Post date: [September 29, 2025, 1:42pm UTC](https://discuss.elastic.co/t/no-rollover-with-data-stream-lifecycle/382165/10 "2025-09-29T13:42:18Z")
I was not able to reproduce it by using your exact template, but I was able to see something similar if I changed the template after I created the data stream, if that’s what you’re doing? The problem is that the retention is an aspect of the data stream itself, not the indices. Changes to the template impact indices that are created on rollover, but never the data stream. To change its retention you have to use the [update data stream lifecycle API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-data-lifecycle). But it sounds like your problem was different, and you’ve figured it out?
---
### Author:  [@mikehaertl](https://discuss.elastic.co/u/mikehaertl)
#### Post date: [September 29, 2025, 3:35pm UTC](https://discuss.elastic.co/t/no-rollover-with-data-stream-lifecycle/382165/11 "2025-09-29T15:35:38Z")
> [@Keith\_Massey](#):
>
> But it sounds like your problem was different, and you’ve figured it out?
I’m still investigating. I’ve now definitely changed the retention time in my `logs@custom` template to `2d` and I’ve deleted all data streams. So I will observe over the next few days.
Actually I can’t see that I do anything unusual but still have not seen once that it behaves as it should. Our setup is quite fresh and nothing fancy IMO.
Is there anything else I can check to find out what could cause it?
---
### Author:  [@Keith\_Massey](https://discuss.elastic.co/u/Keith_Massey)
#### Post date: [September 30, 2025, 1:40pm UTC](https://discuss.elastic.co/t/no-rollover-with-data-stream-lifecycle/382165/12 "2025-09-30T13:40:19Z")
We're you able to make the log level changes I recommended? In your case it would be like:
```auto
PUT /_cluster/settings{
"persistent": {
"logger.org.elasticsearch.datastreams.lifecycle.DataStreamLifecycleService": "TRACE",
"logger.org.elasticsearch.action.admin.indices.rollover.TransportRolloverAction": "TRACE"
}
}
```
See [Update Elasticsearch logging levels | Elastic Docs](https://www.elastic.co/docs/deploy-manage/monitor/logging-configuration/update-elasticsearch-logging-levels) .
---
### Author:  [@mikehaertl](https://discuss.elastic.co/u/mikehaertl)
#### Post date: [September 30, 2025, 1:51pm UTC](https://discuss.elastic.co/t/no-rollover-with-data-stream-lifecycle/382165/13 "2025-09-30T13:51:56Z")
Yes. I already added the log output in a former message, see here:
[https://discuss.elastic.co/t/no-rollover-with-data-stream-lifecycle/382165/3?u=mikehaertl](https://discuss.elastic.co/t/no-rollover-with-data-stream-lifecycle/382165/3)