# Index rollover ealier than described in ILM index lifecycle management
**URL:** https://discuss.elastic.co/t/index-rollover-ealier-than-described-in-ilm-index-lifecycle-management/337996
**Category:** Elasticsearch
**Tags:** ilm-index-lifecycle-management, datastreams
**Created:** [July 10, 2023, 7:53am UTC](https://discuss.elastic.co/t/index-rollover-ealier-than-described-in-ilm-index-lifecycle-management/337996 "2023-07-10T07:53:13Z")
**Posts on this page:** 3
**Page:** 1
### Author:  [@VietDuc](https://discuss.elastic.co/u/VietDuc)
#### Post date: [July 10, 2023, 7:53am UTC](https://discuss.elastic.co/t/index-rollover-ealier-than-described-in-ilm-index-lifecycle-management/337996/1 "2023-07-10T07:53:13Z")
Hi everyone,
I have setup a TSDS with following ILM in ES 8.8.2
```auto
GET .ds-micrometer-metrics-2023.07.08-000036/_ilm/explain
```
```auto
".ds-micrometer-metrics-2023.07.08-000036": {
"index": ".ds-micrometer-metrics-2023.07.08-000036",
"managed": true,
"policy": "metrics-timeseries-datastream",
"time_since_index_creation": "1.72d",
"lifecycle_date_millis": 1688950910593,
"age": "6.61h",
"phase": "hot",
"phase_execution": {
"policy": "metrics-timeseries-datastream",
"phase_definition": {
"min_age": "0ms",
"actions": {
"rollover": {
"max_age": "30d",
"max_primary_shard_size": "25gb"
},
"set_priority": {
"priority": 100
}
}
},
"version": 1,
"modified_date_in_millis": 1683866730933
}
}
}
```
Get stats of the index
```auto
GET .ds-micrometer-metrics-2023.07.08-000036/_stats
```
```auto
".ds-micrometer-metrics-2023.07.08-000036": {
"uuid": "X042z3Z9QaS4BAYBmRCd0Q",
"health": "green",
"status": "open",
"primaries": {
"docs": {
"count": 212237826,
"deleted": 0
},
"shard_stats": {
"total_count": 1
},
```
And
```auto
GET _cat/indices/.ds-micrometer-metrics-2023.07.08-000036?v
```
```auto
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .ds-micrometer-metrics-2023.07.08-000036 X042z3Z9QaS4BAYBmRCd0Q 1 1 212237826 0 11.7gb 5.7gb
```
As you can see, the "time\_since\_index\_creation" is only "1.72d" and the size is only "5.7G" which is way smaller than specified by ILM - 30 days or 25G. There is only 1 shard.
Do you know why this happens?
Before this index, we did a manual "rollover", and the issue only happened after that "rollover", could it be the case?
Do you know how to fix this?
Best regards,
---
### Author:  [@VietDuc](https://discuss.elastic.co/u/VietDuc)
#### Post date: [July 19, 2023, 3:29am UTC](https://discuss.elastic.co/t/index-rollover-ealier-than-described-in-ilm-index-lifecycle-management/337996/2 "2023-07-19T03:29:22Z")
Today we know that it is actually a feature to make sure the performance better
>
>
> \### Description
>
> Elasticsearch ships with some built-in ILM policies, e.g. the…re is one called \`30-days-default\` that looks like this:
>
> \`\`\`json
> {
> "phases": {
> "hot": {
> "actions": {
> "rollover": {
> "max\_primary\_shard\_size": "50gb",
> "max\_age": "30d"
> }
> }
> },
> "warm": {
> "min\_age": "2d",
> "actions": {
> "shrink": {
> "number\_of\_shards": 1
> },
> "forcemerge": {
> "max\_num\_segments": 1
> }
> }
> },
> "delete": {
> "min\_age": "30d",
> "actions":{
> "delete": {}
> }
> }
> },
> "\_meta": {
> "description": "built-in ILM policy using the hot and warm phases with a retention of 30 days",
> "managed": true
> }
> }
> \`\`\`
>
> I would suggest adding a \`max\_primary\_shard\_docs\` next to the \`max\_primary\_shard\_size\`. The actual value needs more discussions, but I'm thinking of something in the order of 200M in order to get a better search experience with space-efficient datasets. So datasets where documents take more than 50GB/200M = 268 bytes would rollover at 50GB while more space-efficient datasets would rollover at 200M documents. My motivation is the following:
> - While shards could rollover before reaching 50GB, they would still not be small: 200M docs is not a small number of documents for a shard. As a data point, shards have a hard limit of 2B docs.
> - Search performance depends more directly on the number of docs than on byte size, so having more control on the number of docs of a shard would give stronger guarantees on the sort of worst-case latency that shards can provide. This is especially relevant given recent/ongoing efforts to improve the space efficiency of Elasticsearch with runtime fields, doc-value-only fields or synthetic source.
> - Aggregations have optimizations for the case when the range filter rewrites to a \`match\_all\` query. Bounding the number of docs per primary shard makes it a bit more likely that some shards fully match the query, and the fact that shards that partially match the range filter have a bounded number of docs also helps bound tail latencies.
> - Rollups can only operate or a rolled-over index, so users cannot enjoy the query speedup of rollups until the primary shard size is reached. Bounding the size of primary shards in a way that makes it easier to reason about query performance helps there too.
> - Many use-cases for Elasticsearch involve \`terms\` or \`composite\` aggregations that need to build global ordinals under the hood. Global ordinals need to be built for the entire shard even though the query might only match a tiny part of it. Bounding the number of docs in a shard also helps bound the time it takes to build global ordinals.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 16, 2023, 3:30am UTC](https://discuss.elastic.co/t/index-rollover-ealier-than-described-in-ilm-index-lifecycle-management/337996/3 "2023-08-16T03:30:12Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.