Elasticsearch ILM rollover from hot to warm phase not happening even when the condition for rollover is met

Elastic Stack Elasticsearch
1

Elasticsearch version 7.1

I have created ILM policy as per which if the index grows more than 50 GB in size or is more than 15days old the index should rollover to warm phase. In my env, this is not happening consistently. I see few old indexes which have grown to more than 100-200GB and then rollover happened. Looking at that I changed the "indices . lifecycle . poll_interval": "1m", it has been a few days since I changed the poll_interval to 1m, yet I see indexes about 80GB in size in Hot phase. I want to understand why the rollover isn't happening when 50 GB size is reached?

ILM:
{
"policy": {
"phases": {
"hot": {
"min_age": "0ms",
"actions": {
"rollover": {
"max_age": "15d",
"max_size": "50gb"
},
"set_priority": {
"priority": 5000
}
}
},
"warm": {
"min_age": "0ms",
"actions": {
"allocate": {
"number_of_replicas": 1,
"include": {},
"exclude": {},
"require": {
"data": "warm"
}
},
"forcemerge": {
"max_num_segments": 1
},
"set_priority": {
"priority": 2000
}
}
},
"delete": {
"min_age": "75d",
"actions": {
"delete": {}
}
}
}
}
}

image

2

Hi ElkNovice,

Can you provide the output of the following:

GET /<your-index>/_ilm/explain?human

That will hopefully shed some light on why it is not rolling over.

3

Output of "GET /myindex/_ilm/explain?human"
{
"indices" : {
"logs-2019.07.28-000001" : {
"index" : "logs-2019.07.28-000001",
"managed" : true,
"policy" : "ilm-15days50gbHot-90daysDelete",
"lifecycle_date" : "2019-07-28T03:37:20.642Z",
"lifecycle_date_millis" : 1564285040642,
"phase" : "hot",
"phase_time" : "2019-07-28T03:37:26.102Z",
"phase_time_millis" : 1564285046102,
"action" : "rollover",
"action_time" : "2019-07-28T03:43:10.006Z",
"action_time_millis" : 1564285390006,
"step" : "check-rollover-ready",
"step_time" : "2019-07-28T03:43:10.006Z",
"step_time_millis" : 1564285390006,
"phase_execution" : {
"policy" : "ilm-15days50gbHot-90daysDelete",
"phase_definition" : {
"min_age" : "0ms",
"actions" : {
"rollover" : {
"max_size" : "50gb",
"max_age" : "15d"
},
"set_priority" : {
"priority" : 5000
}
}
},
"version" : 2,
"modified_date" : "2019-06-11T05:23:23.374Z",
"modified_date_in_millis" : 1560230603374
}
}
}
}

4

Okay the age has definitely not been met (since the index is only a couple of days old), what is the output of

GET /_cat/shards/<your-index>?v

That should show the size of the shards for it

5

index shard prirep state docs store ip node
logs-2019.07.28-000001 0 r STARTED 154154419 44.5gb xxx node1_data_hot
logs-2019.07.28-000001 0 p STARTED 154154419 44.5gb xxx node2_data_hot

6

Is the size in ILM specific to primary shard size?

7

Yes that's correct, since the primary shard is only 44.55gb, it hasn't hit the 50gb limit, that is why this hasn't rolled over yet.

See: https://www.elastic.co/guide/en/elasticsearch/reference/7.3/indices-rollover-index.html which mentions: max_size The maximum estimated size of the primary shard of the index

1 Like
8

I overlooked that when reading the documentation, thanks for your help. Will keep an eye on the primary shard size and validate that rollover happens.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.