Index don't want deleted automatically

Elastic Stack Elasticsearch
1

Hi there,

i wanna ask about index and retention. so here is the question

in my Elasticsearch i have created index automatically and give them policy to go away after 5 days. the problem is they don't want to be deleted automatically. when i check GET _cat/indices?v I can still see the index from 5 months ago. What do you think caused the incident?. please let me know, your help will mean a lot. thanks

2

Please share your ILM policy and an explain on it as well.

3

data1
data1454×507 13.9 KB

data2
data21034×237 16.4 KB

i just enabled the delete phase. and the other i disabled it

4

FYI i use elastic 7.7.1

5

7.7 is EOL and you need to upgrade as a matter of urgency.

I know there was also a bug in older versions where it wouldn't properly set the delete, so upgrading will rule that out.

6

can you give me bug reference? so that there is concrete evidence so i can give it to my team

7
8

Thank you very much

9

How about 7.17? is this bug still exist?

10

No, there it is fixed. 7.17 is the version I would recommend you upgrade to.

11

hmmm i don't think so. my team already install 7.17 version about 2 months with retention: 5 days too. but index from a month before still exist
gambar

gambar
gambar842×204 10.4 KB

12

any suggest?

13

What does the 5-days-default lifecycle policy look like? The fix I linked to was related to how Kibana creates ILM policies. If you have created it using old version or manually you might still have an incorrect policy in place.

14

Here is 5-days-default policy look like

{
  "5-days-default" : {
    "version" : 1,
    "modified_date" : "2022-05-18T02:18:13.088Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_primary_shard_size" : "50gb",
              "max_age" : "30d"
            },
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "delete" : {
          "min_age" : "5d",
          "actions" : {
            "delete" : {
              "delete_searchable_snapshot" : true
            }
          }
        }
      }
    },
    "in_use_by" : {
      "indices" : [
"much index......"
      ],
      "data_streams" : [ ],
      "composable_templates" : [
        "metric_ocp4_prod_esb",
        "ocp4-index_ngrs",
        "metric_ocp4_index_ngrs",
        "ocp4-prod-esb"
      ]
    }
  }
}
17

Hello,
You are using data stream, can you show the output section regarding the index target of your beat/logstash ?
The weird thing is that your policy has a rollover every 30d (or when the primary shard is 50gb) but your backing indices are new every day. That is why i'm asking the previous question.

Regarding the ilm policy itself, the "max_age" inside the hot rollover phase should be 5d instead of 30d, and the "min_age" inside the delete phase should be 0d. This way the indices will do a rollover after 5 days and be deleted right after. The rollover is used to create a new write index, when using data stream you don't have to create aliases it's automaticaly configured.

18

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.