Rollover action during a reindex

Elastic Stack Elasticsearch
, ,
1

Hello,

Currently working on reindexing an old index with a massive 170ishGB single primary shard into a new datastream index, with an ILM policy of 30GB/30D.
We configured for 8 primary, 1 replica, but currently the rollover from ILM policy settings doesn't seem to be taking effect as the reindex is running, but it's still on the initial index:

".ds-xxxxx-2024.09.19-000001"
Primary storage size 117.48gb

The primary storage should cap at our 30GB and process a rollover.
The question I have is does a rollover not occur until a reindex has completed first? I would have thought the rollover would actively take place even during a reindex. I am also unsure about the below, if that's indicating it's in the process of doing rollover, but needs to wait until the reindex has completed first.

Running GET /_ilm/explain:
"indices": {
".ds-xxxxx-2024.09.19-000001": {
"index": ".ds-xxxxx-2024.09.19-000001",
"managed": true,
"policy": "xx-365d",
"index_creation_date_millis": 1726728910258,
"time_since_index_creation": "12.54h",
"lifecycle_date_millis": 1726728910258,
"age": "12.54h",
"phase": "hot",
"phase_time_millis": 1726728911592,
"action": "rollover",
"action_time_millis": 1726728911792,
"step": "check-rollover-ready",
"step_time_millis": 1726728911792,
"phase_execution": {
"policy": "xx-365d",
"phase_definition": {
"min_age": "0ms",
"actions": {
"rollover": {
"max_age": "30d",
"max_primary_shard_size": "30gb"
}
}
},
"version": 1,
"modified_date_in_millis": 1725406131184
}

Thanks in advance!

2

This is the index size, right?

ILM rollovers consider the shard size, not the total index size, it will rollover once one of the primary shards reach the specified size, in this case 30 GB.

Since you set the index to have 8 primary shard, a 117 GB would result in something around 15 GB per shard, which will not trigger the rollover.

3

Ahhh okay, so it's a misunderstanding on my part. Then in that case,
"actions": {
"rollover": {
"max_age": "30d",
"max_primary_shard_size": "30gb"
is indicative of all individual primary shards hitting 30gb on a singular backing index, and only rolls over to a new backing index once they all are at 30gb per primary shard. Correct?

4

Not exactly, when one of the primary shards reaches 30 GB, the rollover will be triggered, not all shards will have the same size, but if any of them reaches the max_primary_shard_size configured value, the rollover can be triggered.

In the documentation you have this mentioned:

max_primary_shard_size
(Optional, byte units) Triggers rollover when the largest primary shard in the index reaches a certain size. This is the maximum size of the primary shards in the index. As with max_size, replicas are ignored.

There is the option max_size, that considers the size of the index, but this is deprecated and should not be used anymore as it will be removed in a future version.

1 Like
5

Great, thank you very much! Sounds like we can close this post