Applying ILM Policy but data not getting deleted

[vikramaddagulla]

Hello,

I am working on configuring an ILM policy to delete any data in an index which is older than 30 days / more than 50GB.

I have an index which has data about 97GB at the moment.

I have configured an alias for this index as shown in the above image..

I have configured a ILM policy to perform rollover of index if any data is more than 50GB and then delete the data after 10 days...

Below is the setting of the same...

I have then applied the above policy to the index which has data of 97GB.

However, even after applying the policy, I do not see the data being deleted from the index...

Could someone please let me know if the above configuration is correct or if I am missing anything ?

[Christian_Dahlqvist]

The ILM setting refers to primary shard size. The size you show include a replica so the primary shard is probably not 50GB yet as the total size is less than 100GB.

[vikramaddagulla]

Ok....Thanks for the information...

Right now, if I refresh the index page, I see an error for the index...The error is as below :

illegal_argument_exception: index name [ohsaccesslogsindex] does not match pattern '^.*-\d+$'

Any action needed for this ?

Please note that : I am trying to configure ILM for an already existing old index...

[Christian_Dahlqvist]

The index the error is referring to does not seem to be a time-based one. ILM deletes complete indices so is usually used with time-based indices.

[vikramaddagulla]

Ok...Does it mean I cannot apply ILM policies to a non-time based index ?

If yes, how can I proceed if I want to delete data from a non-time based index ?

[Christian_Dahlqvist]

You will need to set up and trigger delete by query operations yourself periodically.

[vikramaddagulla]

Ok...Thanks...I was under an assumption that ILM can be applied to non-time based indices...

Could you please share some documentation on how to set up the trigger and delete the data based on custom requirements...??

Thanks in advance...

[Christian_Dahlqvist]

One way is to create a script that is triggered from from. I do not know of any docs showing this.

[vikramaddagulla]

I had seem couple of blogs and came across the below which will delete eveything older than 2016-02-29

Could you please confirm if this would work or if any changes to be made to this ?

DELETE index_name/_query
{
  "query": {
    "filtered": {
      "query": {
        "query_string": {
          "query": "*"
        }
      },
      "filter": {
        "range": {
          "@timestamp": {
            "lte": "2016-02-29"
          }
        }
      }
    }
  }
}

[vikramaddagulla]

Hello,

How do I make sure that my index is a time-series index ?

Right now, I configured the logstash output section to create index in a certain way

( for ex : index ==> "logfileindex" )

This has created the index named logfileindex...But I am unable to apply ILM to this since it is a non-time series based index...

How can I convert this index to a time-series based ?

If it cannot be converted, how do I create a time-series based index with my current settings ?