Elasticsearch ILM bug how to debug?

ILM policy is set: rollover daily / delete after 31days.
Index is deleted after 7days (ignoring 31days setup)
I tried to change the value 30,31 and no change.
This ILM is used by 2 templates and 2 indexes. test_xxx, and production_xxx
ILM works on test but does not work on production.
Is this a BUG?
How can I troufix it?

Policy conf:

{
"sys_metrics_mediator_policy" : {
"version" : 2,
"modified_date" : "2020-03-13T16:00:58.131Z",
"policy" : {
"phases" : {
"hot" : {
"min_age" : "0ms",
"actions" : {
"rollover" : {
"max_size" : "20gb",
"max_age" : "1d"
}
}
},
"delete" : {
"min_age" : "31d",
"actions" : {
"delete" : { }
}
}
}
}
}
}

from the list see 31 test indices available and only 7 from production

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open sys_metrics_mediator-000033 UVdiNAguRwOp2uEHuzGyXA 1 1 25281 0 6.2mb 3.1mb !!! 5,5 days old
green open sys_metrics_mediator-000034 AEPi-psQQBqrFwRQxn5_4w 1 1 27136 0 6.5mb 3.2mb
green open sys_metrics_mediator-000035 jdLIFzQISL21o20rmQ0Ulg 1 1 18071 0 4.5mb 2.2mb
green open sys_metrics_mediator-000036 rglrNih4Q7OAUDosbNikhA 1 1 27100 0 7mb 3.5mb
green open sys_metrics_mediator-000037 Gso-nCZVQ0mE_cXAdkwn6g 1 1 29884 0 7.4mb 3.6mb
green open sys_metrics_mediator-000038 taiksPNEQBGzFcDh_wrCgw 1 1 70589 0 15.6mb 7.8mb
green open sys_metrics_mediator-000039 sm4R2yleR3eCUFLOHHb_hQ 1 1 12424 0 2.9mb 1.4mb
green open test_sys_metrics_mediator-000008 trSjtdAlS1WWh_Ul7P51og 1 0 0 0 283b 283b !!! 30.5 days old
green open test_sys_metrics_mediator-000009 H6XmmOBNTxSY5clsvd5lIw 1 0 0 0 283b 283b
green open test_sys_metrics_mediator-000010 F-67mg0_T1ihEhA231gvGg 1 0 0 0 283b 283b
green open test_sys_metrics_mediator-000011 5QAuNUL7T42PNT3RX97H7A 1 0 0 0 283b 283b
green open test_sys_metrics_mediator-000012 gg8Or-OUTbedH9wh7VaSnQ 1 0 0 0 283b 283b
green open test_sys_metrics_mediator-000013 V2WDJQYwQrONdKLdLD7ZHw 1 0 0 0 283b 283b
green open test_sys_metrics_mediator-000014 OEPW13wiQQaoBwfkeHI08w 1 0 0 0 283b 283b
green open test_sys_metrics_mediator-000015 sB_7Ta_CSzSbFOJpmlDlew 1 0 0 0 283b 283b
green open test_sys_metrics_mediator-000016 UYwkVhVlSN-oLOKUwYgkGw 1 0 0 0 283b 283b
green open test_sys_metrics_mediator-000017 80tEWaoyRJCrV8eSXfO6ww 1 0 0 0 283b 283b
green open test_sys_metrics_mediator-000018 peG042KgTje_IHlGXQShXQ 1 0 0 0 283b 283b
green open test_sys_metrics_mediator-000019 4WfN88gmTFChusEu-cq8ew 1 0 0 0 283b 283b
green open test_sys_metrics_mediator-000020 VfaBp96eQm6XvNjiWMrB8Q 1 0 0 0 283b 283b
green open test_sys_metrics_mediator-000023 _pID3bp1Thq4NKFc-6mZ3Q 1 0 0 0 283b 283b
green open test_sys_metrics_mediator-000024 gL9jHsjjTOartuEjUdyrcg 1 0 0 0 283b 283b
green open test_sys_metrics_mediator-000025 v_2yFlYbRESWPwm4HPlSow 1 0 0 0 283b 283b
green open test_sys_metrics_mediator-000026 TETbBz93QOi7yRtvx80Ibg 1 0 0 0 283b 283b
green open test_sys_metrics_mediator-000027 pt81Ri00S1qrV7jlB6aHKw 1 0 0 0 283b 283b
green open test_sys_metrics_mediator-000028 vqHmQc98S_S-AiIW43oClg 1 0 0 0 283b 283b

prod index ilm explain - see compared to test ilm explain below - they are the same

{
"indices" : {
"sys_metrics_mediator-000033" : {
"index" : "sys_metrics_mediator-000033",
"managed" : true,
"policy" : "sys_metrics_mediator_policy",
"lifecycle_date_millis" : 1583876910758,
"age" : "5.34d",
"phase" : "hot",
"phase_time_millis" : 1583789910960,
"action" : "complete",
"action_time_millis" : 1583876911407,
"step" : "complete",
"step_time_millis" : 1583876911407,
"phase_execution" : {
"policy" : "sys_metrics_mediator_policy",
"phase_definition" : {
"min_age" : "0ms",
"actions" : {
"rollover" : {
"max_size" : "20gb",
"max_age" : "1d"
}
}
},
"version" : 1,
"modified_date_in_millis" : 1581004810207
}
}
}
}

test index ilm explain

{
"indices" : {
"test_sys_metrics_mediator-000008" : {
"index" : "test_sys_metrics_mediator-000008",
"managed" : true,
"policy" : "sys_metrics_mediator_policy",
"lifecycle_date_millis" : 1581702510778,
"age" : "30.51d",
"phase" : "hot",
"phase_time_millis" : 1581615510983,
"action" : "complete",
"action_time_millis" : 1581702511439,
"step" : "complete",
"step_time_millis" : 1581702511439,
"phase_execution" : {
"policy" : "sys_metrics_mediator_policy",
"phase_definition" : {
"min_age" : "0ms",
"actions" : {
"rollover" : {
"max_size" : "20gb",
"max_age" : "1d"
}
}
},
"version" : 1,
"modified_date_in_millis" : 1581004810207
}
}
}
}

Template:

{
"sys_metrics_mediator_template" : {
"order" : 0,
"index_patterns" : [
"sys_metrics_mediator*"
],
"settings" : {
"index" : {
"lifecycle" : {
"name" : "sys_metrics_mediator_policy",
"rollover_alias" : "sys_metrics_mediator"
},
"number_of_shards" : "1",
"number_of_replicas" : "1"
}
},...

was set to 7days and changed to 30Days later.
New indexes created are probably not accepting the changed value.
Do I have to refresh something or is this a BUG?
I have a version 7.4.2

1 Like