Index rollover ealier than described in ILM index lifecycle management

VietDuc · July 10, 2023, 7:53am

Hi everyone,
I have setup a TSDS with following ILM in ES 8.8.2

GET .ds-micrometer-metrics-2023.07.08-000036/_ilm/explain

 ".ds-micrometer-metrics-2023.07.08-000036": {
      "index": ".ds-micrometer-metrics-2023.07.08-000036",
      "managed": true,
      "policy": "metrics-timeseries-datastream",
      "time_since_index_creation": "1.72d",
      "lifecycle_date_millis": 1688950910593,
      "age": "6.61h",
      "phase": "hot",
      "phase_execution": {
        "policy": "metrics-timeseries-datastream",
        "phase_definition": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_age": "30d",
              "max_primary_shard_size": "25gb"
            },
            "set_priority": {
              "priority": 100
            }
          }
        },
        "version": 1,
        "modified_date_in_millis": 1683866730933
      }
    }
  }

Get stats of the index

GET .ds-micrometer-metrics-2023.07.08-000036/_stats

   ".ds-micrometer-metrics-2023.07.08-000036": {
      "uuid": "X042z3Z9QaS4BAYBmRCd0Q",
      "health": "green",
      "status": "open",
      "primaries": {
        "docs": {
          "count": 212237826,
          "deleted": 0
        },
        "shard_stats": {
          "total_count": 1
        },

And

GET _cat/indices/.ds-micrometer-metrics-2023.07.08-000036?v

health status index                                    uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .ds-micrometer-metrics-2023.07.08-000036 X042z3Z9QaS4BAYBmRCd0Q   1   1  212237826            0     11.7gb          5.7gb

As you can see, the "time_since_index_creation" is only "1.72d" and the size is only "5.7G" which is way smaller than specified by ILM - 30 days or 25G. There is only 1 shard.

Do you know why this happens?

Before this index, we did a manual "rollover", and the issue only happened after that "rollover", could it be the case?

Do you know how to fix this?

Best regards,

VietDuc · July 19, 2023, 3:29am

Today we know that it is actually a feature to make sure the performance better

github.com/elastic/elasticsearch

Add a `max_primary_shard_docs` to default ILM policies

opened 12:11PM - 31 May 22 UTC

closed 01:21PM - 08 Mar 23 UTC

jpountz

>enhancement :Data Management/ILM+SLM Team:Data Management

### Description Elasticsearch ships with some built-in ILM policies, e.g. the…re is one called `30-days-default` that looks like this: ```json { "phases": { "hot": { "actions": { "rollover": { "max_primary_shard_size": "50gb", "max_age": "30d" } } }, "warm": { "min_age": "2d", "actions": { "shrink": { "number_of_shards": 1 }, "forcemerge": { "max_num_segments": 1 } } }, "delete": { "min_age": "30d", "actions":{ "delete": {} } } }, "_meta": { "description": "built-in ILM policy using the hot and warm phases with a retention of 30 days", "managed": true } } ``` I would suggest adding a `max_primary_shard_docs` next to the `max_primary_shard_size`. The actual value needs more discussions, but I'm thinking of something in the order of 200M in order to get a better search experience with space-efficient datasets. So datasets where documents take more than 50GB/200M = 268 bytes would rollover at 50GB while more space-efficient datasets would rollover at 200M documents. My motivation is the following: - While shards could rollover before reaching 50GB, they would still not be small: 200M docs is not a small number of documents for a shard. As a data point, shards have a hard limit of 2B docs. - Search performance depends more directly on the number of docs than on byte size, so having more control on the number of docs of a shard would give stronger guarantees on the sort of worst-case latency that shards can provide. This is especially relevant given recent/ongoing efforts to improve the space efficiency of Elasticsearch with runtime fields, doc-value-only fields or synthetic source. - Aggregations have optimizations for the case when the range filter rewrites to a `match_all` query. Bounding the number of docs per primary shard makes it a bit more likely that some shards fully match the query, and the fact that shards that partially match the range filter have a bounded number of docs also helps bound tail latencies. - Rollups can only operate or a rolled-over index, so users cannot enjoy the query speedup of rollups until the primary shard size is reached. Bounding the size of primary shards in a way that makes it easier to reason about query performance helps there too. - Many use-cases for Elasticsearch involve `terms` or `composite` aggregations that need to build global ordinals under the hood. Global ordinals need to be built for the entire shard even though the query might only match a tiny part of it. Bounding the number of docs in a shard also helps bound the time it takes to build global ordinals.

system · August 16, 2023, 3:30am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Inconsistent datastream index rollover Elasticsearch ilm-index-lifecycle-management , datastreams	1	343	February 27, 2023
ILM - Does not trigger rollover Elasticsearch ilm-index-lifecycle-management	8	648	January 4, 2024
Index Rollover not working Elasticsearch ilm-index-lifecycle-management	4	2563	June 26, 2020
Issue with index rollover via ILM Elasticsearch ilm-index-lifecycle-management	1	431	February 14, 2020
ILM policy not working Elasticsearch ilm-index-lifecycle-management	2	5208	June 17, 2019

Index rollover ealier than described in ILM index lifecycle management

Related topics