Indices keep getting deleted automatically

Elastic Stack Elasticsearch
1

We are running ELK 7.6.2 stack.

It seems that indices are getting deleted / cleaned automatically by ES.

We have ILM policy of 2 days retention for the index, but somehow it deletes the current one too.

After enabling debug in both Logstash and ES, the following was seen:

[2022-04-28T16:39:22,027][DEBUG][o.e.i.c.IndicesClusterStateService] [xx-xxx-xxx] [[prd_tv_global_2022.04.22/wewfo9p-TV6wK9fuLZf7ig]] cleaning index, no longer part of the metadata

[2022-04-28T16:39:22,355][DEBUG][o.e.i.c.IndicesClusterStateService] [xx-xxx-xxx] [[prd_tv_global_2022.04.25/Hp-K_6r6SSqPMayCGWsi1w]] cleaning index, no longer part of the metadata

Note todays' timestamp of April 28:

[2022-04-28T16:39:22,425][DEBUG][o.e.i.c.IndicesClusterStateService] [xx-xxx-xxx] [[prd_tv_global_2022.04.28/uYbaWQDgTwiDgrGOgIvcJA]] cleaning index, no longer part of the metadata

stdout was enabled in the Logstash config and messages could be seen coming into Logstash but index is not getting created.

output {
 file {
   path => "/opt/gtal/ptal/elasticsearch/logs/Logstash/global_rubydebug.txt"
   codec => rubydebug
 }

Please guide. Please note that I took out the index (in question) from index template too.

2

It'd be useful if you could show us the policy.

3 
{
  "cleanup-crnlp" : {
    "version" : 1,
    "modified_date" : "2021-03-02T20:05:41.551Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : { }
        },
        "delete" : {
          "min_age" : "2d",
          "actions" : {
            "delete" : { }
          }
        }
      }
    }
  }
}
4

The related index was part of the "2days_crnlp_template" and I took it out today just to be sure.

GET _cat/templates/2days_crnlp_template
5

Hi @warkolm. Removed the entire template just to be sure but somehow index is not getting created. The Logstash debug shows messages coming in at a very very fast pace.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.