Applying ILM Policy but data not getting deleted

Elastic Stack Elasticsearch
1

Hello,

I am working on configuring an ILM policy to delete any data in an index which is older than 30 days / more than 50GB.

I have an index which has data about 97GB at the moment.

image
image911×382 25.7 KB

I have configured an alias for this index as shown in the above image..

I have configured a ILM policy to perform rollover of index if any data is more than 50GB and then delete the data after 10 days...

Below is the setting of the same...

image
image1062×516 31 KB

image
image1120×206 15.2 KB

I have then applied the above policy to the index which has data of 97GB.

However, even after applying the policy, I do not see the data being deleted from the index...

Could someone please let me know if the above configuration is correct or if I am missing anything ?

2

The ILM setting refers to primary shard size. The size you show include a replica so the primary shard is probably not 50GB yet as the total size is less than 100GB.

3

Ok....Thanks for the information...

Right now, if I refresh the index page, I see an error for the index...The error is as below :

illegal_argument_exception: index name [ohsaccesslogsindex] does not match pattern '^.*-\d+$'

Any action needed for this ?

Please note that : I am trying to configure ILM for an already existing old index...

4

The index the error is referring to does not seem to be a time-based one. ILM deletes complete indices so is usually used with time-based indices.

5

Ok...Does it mean I cannot apply ILM policies to a non-time based index ?

If yes, how can I proceed if I want to delete data from a non-time based index ?

6

You will need to set up and trigger delete by query operations yourself periodically.

7

Ok...Thanks...I was under an assumption that ILM can be applied to non-time based indices...

Could you please share some documentation on how to set up the trigger and delete the data based on custom requirements...??

Thanks in advance...

8

One way is to create a script that is triggered from from. I do not know of any docs showing this.

9

I had seem couple of blogs and came across the below which will delete eveything older than 2016-02-29

Could you please confirm if this would work or if any changes to be made to this ?

DELETE index_name/_query
{
  "query": {
    "filtered": {
      "query": {
        "query_string": {
          "query": "*"
        }
      },
      "filter": {
        "range": {
          "@timestamp": {
            "lte": "2016-02-29"
          }
        }
      }
    }
  }
}
10

Hello,

How do I make sure that my index is a time-series index ?

Right now, I configured the logstash output section to create index in a certain way

( for ex : index ==> "logfileindex" )

This has created the index named logfileindex...But I am unable to apply ILM to this since it is a non-time series based index...

How can I convert this index to a time-series based ?

If it cannot be converted, how do I create a time-series based index with my current settings ?

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.