ILM policy deletes index after 7 days despite the configuration is set to delete after 30days

I have a problem with IML policy, it deletes index after 7 days despite the configuration is set to delete after 30days.
What I have configured wrong.? (see all detail below)

  1. I have created index templa
  2. created index policy
  3. assigned policy to template (with correct alias)
  4. created empty index with alias
  5. start import of the data

I found data are deleted after 7 days, bud policy is set delete 30days.
I do not use warm phase (is this a problem?)

the policy configuration:

GET _ilm/policy/datasource_metrics_mediator_policy
{
"datasource_metrics_mediator_policy" : {
"version" : 1,
"modified_date" : "2020-02-06T16:00:10.207Z",
"policy" : {
"phases" : {
"hot" : {
"min_age" : "0ms",
"actions" : {
"rollover" : {
"max_size" : "20gb",
"max_age" : "1d"
}
}
},
"delete" : {
"min_age" : "30d",
"actions" : {
"delete" : { }
}
}
}
}
}
}

From the list I see I have only 7 daily indices - older are deleted

GET _cat/indices/datasourcemetricsmediator*?v&h=index,store.size,creation.date.string&s=index
index store.size creation.date.string
datasource_metrics_mediator-000023 10mb 2020-02-28T19:58:30.661Z
datasource_metrics_mediator-000024 8.6mb 2020-02-29T20:08:30.656Z
datasource_metrics_mediator-000025 5.9mb 2020-03-01T20:18:30.661Z
datasource_metrics_mediator-000026 11.1mb 2020-03-02T20:28:30.810Z
datasource_metrics_mediator-000027 9.5mb 2020-03-03T20:38:30.753Z
datasource_metrics_mediator-000028 5.4mb 2020-03-04T20:48:30.738Z
datasource_metrics_mediator-000029 1.8mb 2020-03-05T20:58:30.775Z

the setting of the index: (confirmaton correct policy is assigned : datasource_metrics_mediator_policy)

GET datasource_metrics_mediator-000023/_settings
{
"datasource_metrics_mediator-000023" : {
"settings" : {
"index" : {
"lifecycle" : {
"name" : "datasource_metrics_mediator_policy",
"rollover_alias" : "datasource_metrics_mediator",
"indexing_complete" : "true"
},
"number_of_shards" : "1",
"provided_name" : "datasource_metrics_mediator-000023",
"creation_date" : "1582919910661",
"number_of_replicas" : "1",
"uuid" : "1ExAlnsTRgue_wBxxeOreg",
"version" : {
"created" : "7040299"
}
}
}
}
}

6days old index ILM explains shows it is 5.6d old so I expect it is not deleted next day. But it is.

GET datasource_metrics_mediator-000023/_ilm/explain
{
"indices" : {
"datasource_metrics_mediator-000023" : {
"index" : "datasource_metrics_mediator-000023",
"managed" : true,
"policy" : "datasource_metrics_mediator_policy",
"lifecycle_date_millis" : 1583006910726,
"age" : "5.6d",
"phase" : "hot",
"phase_time_millis" : 1582919911011,
"action" : "complete",
"action_time_millis" : 1583006911226,
"step" : "complete",
"step_time_millis" : 1583006911226,
"phase_execution" : {
"policy" : "datasource_metrics_mediator_policy",
"phase_definition" : {
"min_age" : "0ms",
"actions" : {
"rollover" : {
"max_size" : "20gb",
"max_age" : "1d"
}
}
},
"version" : 1,
"modified_date_in_millis" : 1581004810207
}
}
}
}

Log from master node - index deleted after 7 days

server04prahkz:/data/elasticsearch/log# zcat company-cz-cem-2020-03* |grep deleting |grep datasource_metrics_mediator
{"type": "server", "timestamp": "2020-03-01T03:13:25,625+01:00", "level": "INFO", "component": "o.e.c.m.MetaDataDeleteIndexService", "cluster.name": "company-cz-cem", "node.name": "server04-prahkz", "message": "[datasource_metrics_mediator-000017/TY2yahxtRIG-dvypEW9HHA] deleting index", "cluster.uuid": "OAIGGQ4QTqiz4i_tgdFx3g", "node.id": "xIiNk12YR6K8ZMfA_Af7JA" }
[2020-03-01T03:13:25,625][INFO ][o.e.c.m.MetaDataDeleteIndexService] [server04-prahkz] [datasource_metrics_mediator-000017/TY2yahxtRIG-dvypEW9HHA] deleting index
{"type": "server", "timestamp": "2020-03-02T03:29:18,405+01:00", "level": "INFO", "component": "o.e.c.m.MetaDataDeleteIndexService", "cluster.name": "company-cz-cem", "node.name": "server04-prahkz", "message": "[datasource_metrics_mediator-000018/PaoX4rIiR6i_A3BI0FtHnQ] deleting index", "cluster.uuid": "OAIGGQ4QTqiz4i_tgdFx3g", "node.id": "xIiNk12YR6K8ZMfA_Af7JA" }
[2020-03-02T03:29:18,405][INFO ][o.e.c.m.MetaDataDeleteIndexService] [server04-prahkz] [datasource_metrics_mediator-000018/PaoX4rIiR6i_A3BI0FtHnQ] deleting index
{"type": "server", "timestamp": "2020-03-03T03:30:20,911+01:00", "level": "INFO", "component": "o.e.c.m.MetaDataDeleteIndexService", "cluster.name": "company-cz-cem", "node.name": "server04-prahkz", "message": "[datasource_metrics_mediator-000019/NefZN677QEOY1FiARSCiRQ] deleting index", "cluster.uuid": "OAIGGQ4QTqiz4i_tgdFx3g", "node.id": "xIiNk12YR6K8ZMfA_Af7JA" }
[2020-03-03T03:30:20,911][INFO ][o.e.c.m.MetaDataDeleteIndexService] [server04-prahkz] [datasource_metrics_mediator-000019/NefZN677QEOY1FiARSCiRQ] deleting index
{"type": "server", "timestamp": "2020-03-04T03:47:12,850+01:00", "level": "INFO", "component": "o.e.c.m.MetaDataDeleteIndexService", "cluster.name": "company-cz-cem", "node.name": "server04-prahkz", "message": "[datasource_metrics_mediator-000020/ORRFfLOnTwyo6lGdyT5Puw] deleting index", "cluster.uuid": "OAIGGQ4QTqiz4i_tgdFx3g", "node.id": "xIiNk12YR6K8ZMfA_Af7JA" }
[2020-03-04T03:47:12,850][INFO ][o.e.c.m.MetaDataDeleteIndexService] [server04-prahkz] [datasource_metrics_mediator-000020/ORRFfLOnTwyo6lGdyT5Puw] deleting index
{"type": "server", "timestamp": "2020-03-05T03:13:28,275+01:00", "level": "INFO", "component": "o.e.c.m.MetaDataDeleteIndexService", "cluster.name": "company-cz-cem", "node.name": "server04-prahkz", "message": "[test_datasource_metrics_mediator-000021/5lCvQk7KSm-oG8pfqg627A] deleting index", "cluster.uuid": "OAIGGQ4QTqiz4i_tgdFx3g", "node.id": "xIiNk12YR6K8ZMfA_Af7JA" }
{"type": "server", "timestamp": "2020-03-05T03:13:28,382+01:00", "level": "INFO", "component": "o.e.c.m.MetaDataDeleteIndexService", "cluster.name": "company-cz-cem", "node.name": "server04-prahkz", "message": "[datasource_metrics_mediator-000021/yT7xOM3hRwqyoe5ab1-6yA] deleting index", "cluster.uuid": "OAIGGQ4QTqiz4i_tgdFx3g", "node.id": "xIiNk12YR6K8ZMfA_Af7JA" }
[2020-03-05T03:13:28,275][INFO ][o.e.c.m.MetaDataDeleteIndexService] [server04-prahkz] [test_datasource_metrics_mediator-000021/5lCvQk7KSm-oG8pfqg627A] deleting index
[2020-03-05T03:13:28,382][INFO ][o.e.c.m.MetaDataDeleteIndexService] [server04-prahkz] [datasource_metrics_mediator-000021/yT7xOM3hRwqyoe5ab1-6yA] deleting index

2 Likes

ILM was set to 7days and changed to 30Days later.
New indexes created are probably not accepting the changed value.
Do I have to refresh something or is this a BUG?
I have a version 7.4.2

2 Likes

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.