Issue with Elasticsearch indexes

Hello everyone, I am new to ELK and the issue I am experiencing is that indexes are after 3 days reduces to 25x bytes and 0 documents. We are viewing index patterns in Kibana but it will display the 3 days only. Nothing before that.
We are running Elasticsearch and Kibana 7.12.1 using docker-compose.
I have setup policy which should keep the data in for at least 10 days but the indexes older then 3 days are reduced to 253bytes and 0 documents
Can someone help.
Thank you

Hello @milank2

Welcome to the elastic community!!!
Could you share us the ilm explain output

  "indices" : {
    "msg-stas-audit-panel-state-2023-4-26" : {
      "index" : "msg-stas-audit-panel-state-2023-4-26",
      "managed" : true,
      "policy" : "msg-stas-audit-policy-policy",
      "lifecycle_date_millis" : 1682467139629,
      "age" : "2.62d",
      "phase" : "new",
      "phase_time_millis" : 1682531366167,
      "action" : "complete",
      "action_time_millis" : 1682531366167,
      "step" : "complete",
      "step_time_millis" : 1682531366167,
      "phase_execution" : {
        "policy" : "msg-stas-audit-policy-policy",
        "version" : 1,
        "modified_date_in_millis" : 1682531059835

Thanks for replying here it is

Hi @milank2 Welcome to the community.

Can you run

GET _cat/indices?v

Something else is going on as there is no ILM action that does this.

Those 0 document 255 byte indices are not "deleted" or cleaned indices they are "stub" indices created by a rollover policy and then data is never written to them.

Please also share the

GET _ilm/policy/msg-stas-audit-policy-policy

Hi Stephenb, here it is:
for GET /_cat/indices?v I only took msg-stas* since it is so big:

GET /_cat/indices/msg-stas-audit-*?v
health status index                                 uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   msg-stas-audit-asdr-message-2023-4-25 X5ZQbbN5Q06i0BgHGYA5hA   1   0          0            0       253b           253b
green  open   msg-stas-audit-panel-state-2023-4-28  5zsirpTzT16jXH2MAu03yg   1   0     554217            0    168.2mb        168.2mb
green  open   msg-stas-audit-asdr-message-2023-4-24 -qg11lz9TMur6itk_Jy3aQ   1   0          0            0       253b           253b
green  open   msg-stas-audit-asdr-message-2023-4-23 1GPreZGzSJC2Hf3weI34tQ   1   0          0            0       253b           253b
green  open   msg-stas-audit-asdr-message-2023-4-22 TLQkFIY8TC-3R30GSPWMog   1   0          0            0       253b           253b
green  open   msg-stas-audit-asdr-message-2023-4-21 B22fHdhTTAOXcsVCicyToA   1   0          0            0       253b           253b
green  open   msg-stas-audit-asdr-message-2023-4-20 xBPpf7JgQaSoMHfCOuXQFA   1   0          0            0       253b           253b
green  open   msg-stas-audit-panel-state-2023-4-25  YXFfV-vGTKqf6UYweQZxZg   1   0          0            0       250b           250b
yellow open   msg-stas-audit-asdr-message-2023-4-28 jVzqwYZQTuGn-cILhGAlyQ   1   1    4913624            0    354.6mb        354.6mb
yellow open   msg-stas-audit-asdr-message-2023-4-27 tMNvpWaVStquVhvQACtXiA   1   1    4327796            0      306mb          306mb
yellow open   msg-stas-audit-panel-state-2023-4-27  PN-qgRcXSWCcED3ZtRd3dQ   1   1     489735            0    142.6mb        142.6mb
green  open   msg-stas-audit-asdr-message-2023-4-26 NoZpcGrVTNajl8MrT4wwCg   1   0    6846149           60    483.7mb        483.7mb
green  open   msg-stas-audit-panel-state-2023-4-26  I7KIUOvFTrOgywbjBsAKdg   1   0     714810            6    207.4mb        207.4mb
green  open   msg-stas-audit-panel-state-2023-1-24  emquHpEySj2Somba5GCw-Q   1   0          0            0       231b           231b

and

  "msg-stas-audit-policy-policy" : {
    "version" : 12,
    "modified_date" : "2023-04-28T14:21:44.028Z",
    "policy" : {
      "phases" : {
        "warm" : {
          "min_age" : "15d",
          "actions" : { }
        },
        "cold" : {
          "min_age" : "30d",
          "actions" : {
            "readonly" : { }
          }
        },
        "hot" : {
          "min_age" : "10d",
          "actions" : {
            "rollover" : {
              "max_size" : "5gb",
              "max_age" : "14d"
            }
          }
        },
        "delete" : {
          "min_age" : "60d",
          "actions" : {
            "delete" : {
              "delete_searchable_snapshot" : true
            }
          }
        }
      }
    }
  }

Hi @milank2

1st that is not correct... that should be

        "min_age": "0ms",

That could be the source of your problem... but not sure

green  open   msg-stas-audit-asdr-message-2023-4-24 -qg11lz9TMur6itk_Jy3aQ   1   0          0            0       253b           253b
green  open   msg-stas-audit-asdr-message-2023-4-23 1GPreZGzSJC2Hf3weI34tQ   1   0          0            0       253b           253b
green  open   msg-stas-audit-asdr-message-2023-4-22 TLQkFIY8TC-3R30GSPWMog   1   0          0            0       253b           253b
green  open   msg-stas-audit-asdr-message-2023-4-21 B22fHdhTTAOXcsVCicyToA   1   0          0            0       253b           253b
green  open   msg-stas-audit-asdr-message-2023-4-20 xBPpf7JgQaSoMHfCOuXQFA   1   0          0            0       253b           253b

Those are created by rollover ... interesting thing is that they have 0 Replicas which make me thing you have another index template that might be matching

First fix the ILM policy ... clean up those empty indices and then observe

Hi Stephenb.
Thanks for replying, so I have changed the policy to "min_age": "0ms"
removed most of the indexes with 0 docs and 25x bytes. Interesting thing was that I have recovered bunch of shards by doing that.
Anyhow Kibana was still displaying only last 3 days, well seeing still the same results I have removed all the policy's and aliases associated with the indexes but the 3 day display in Kibana still remains.

You may have some other issues... Keep an eye out

Welcome to our community! :smiley:

Please note that version is EOL and no longer supported, you should be looking to upgrade as a matter of urgency.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.