Hello everyone, I am new to ELK and the issue I am experiencing is that indexes are after 3 days reduces to 25x bytes and 0 documents. We are viewing index patterns in Kibana but it will display the 3 days only. Nothing before that.
We are running Elasticsearch and Kibana 7.12.1 using docker-compose.
I have setup policy which should keep the data in for at least 10 days but the indexes older then 3 days are reduced to 253bytes and 0 documents
Can someone help.
Thank you
Hello @milank2
Welcome to the elastic community!!!
Could you share us the ilm explain output
"indices" : {
"msg-stas-audit-panel-state-2023-4-26" : {
"index" : "msg-stas-audit-panel-state-2023-4-26",
"managed" : true,
"policy" : "msg-stas-audit-policy-policy",
"lifecycle_date_millis" : 1682467139629,
"age" : "2.62d",
"phase" : "new",
"phase_time_millis" : 1682531366167,
"action" : "complete",
"action_time_millis" : 1682531366167,
"step" : "complete",
"step_time_millis" : 1682531366167,
"phase_execution" : {
"policy" : "msg-stas-audit-policy-policy",
"version" : 1,
"modified_date_in_millis" : 1682531059835
Thanks for replying here it is
Hi @milank2 Welcome to the community.
Can you run
GET _cat/indices?v
Something else is going on as there is no ILM action that does this.
Those 0 document 255 byte indices are not "deleted" or cleaned indices they are "stub" indices created by a rollover policy and then data is never written to them.
Please also share the
GET _ilm/policy/msg-stas-audit-policy-policy
Hi Stephenb, here it is:
for GET /_cat/indices?v I only took msg-stas* since it is so big:
GET /_cat/indices/msg-stas-audit-*?v
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open msg-stas-audit-asdr-message-2023-4-25 X5ZQbbN5Q06i0BgHGYA5hA 1 0 0 0 253b 253b
green open msg-stas-audit-panel-state-2023-4-28 5zsirpTzT16jXH2MAu03yg 1 0 554217 0 168.2mb 168.2mb
green open msg-stas-audit-asdr-message-2023-4-24 -qg11lz9TMur6itk_Jy3aQ 1 0 0 0 253b 253b
green open msg-stas-audit-asdr-message-2023-4-23 1GPreZGzSJC2Hf3weI34tQ 1 0 0 0 253b 253b
green open msg-stas-audit-asdr-message-2023-4-22 TLQkFIY8TC-3R30GSPWMog 1 0 0 0 253b 253b
green open msg-stas-audit-asdr-message-2023-4-21 B22fHdhTTAOXcsVCicyToA 1 0 0 0 253b 253b
green open msg-stas-audit-asdr-message-2023-4-20 xBPpf7JgQaSoMHfCOuXQFA 1 0 0 0 253b 253b
green open msg-stas-audit-panel-state-2023-4-25 YXFfV-vGTKqf6UYweQZxZg 1 0 0 0 250b 250b
yellow open msg-stas-audit-asdr-message-2023-4-28 jVzqwYZQTuGn-cILhGAlyQ 1 1 4913624 0 354.6mb 354.6mb
yellow open msg-stas-audit-asdr-message-2023-4-27 tMNvpWaVStquVhvQACtXiA 1 1 4327796 0 306mb 306mb
yellow open msg-stas-audit-panel-state-2023-4-27 PN-qgRcXSWCcED3ZtRd3dQ 1 1 489735 0 142.6mb 142.6mb
green open msg-stas-audit-asdr-message-2023-4-26 NoZpcGrVTNajl8MrT4wwCg 1 0 6846149 60 483.7mb 483.7mb
green open msg-stas-audit-panel-state-2023-4-26 I7KIUOvFTrOgywbjBsAKdg 1 0 714810 6 207.4mb 207.4mb
green open msg-stas-audit-panel-state-2023-1-24 emquHpEySj2Somba5GCw-Q 1 0 0 0 231b 231b
and
"msg-stas-audit-policy-policy" : {
"version" : 12,
"modified_date" : "2023-04-28T14:21:44.028Z",
"policy" : {
"phases" : {
"warm" : {
"min_age" : "15d",
"actions" : { }
},
"cold" : {
"min_age" : "30d",
"actions" : {
"readonly" : { }
}
},
"hot" : {
"min_age" : "10d",
"actions" : {
"rollover" : {
"max_size" : "5gb",
"max_age" : "14d"
}
}
},
"delete" : {
"min_age" : "60d",
"actions" : {
"delete" : {
"delete_searchable_snapshot" : true
}
}
}
}
}
}
Hi @milank2
1st that is not correct... that should be
"min_age": "0ms",
That could be the source of your problem... but not sure
green open msg-stas-audit-asdr-message-2023-4-24 -qg11lz9TMur6itk_Jy3aQ 1 0 0 0 253b 253b
green open msg-stas-audit-asdr-message-2023-4-23 1GPreZGzSJC2Hf3weI34tQ 1 0 0 0 253b 253b
green open msg-stas-audit-asdr-message-2023-4-22 TLQkFIY8TC-3R30GSPWMog 1 0 0 0 253b 253b
green open msg-stas-audit-asdr-message-2023-4-21 B22fHdhTTAOXcsVCicyToA 1 0 0 0 253b 253b
green open msg-stas-audit-asdr-message-2023-4-20 xBPpf7JgQaSoMHfCOuXQFA 1 0 0 0 253b 253b
Those are created by rollover ... interesting thing is that they have 0 Replicas which make me thing you have another index template that might be matching
First fix the ILM policy ... clean up those empty indices and then observe
Hi Stephenb.
Thanks for replying, so I have changed the policy to "min_age": "0ms"
removed most of the indexes with 0 docs and 25x bytes. Interesting thing was that I have recovered bunch of shards by doing that.
Anyhow Kibana was still displaying only last 3 days, well seeing still the same results I have removed all the policy's and aliases associated with the indexes but the 3 day display in Kibana still remains.
You may have some other issues... Keep an eye out
Welcome to our community! 
Please note that version is EOL and no longer supported, you should be looking to upgrade as a matter of urgency.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.