Index templates lifecycle management

[Craig2188 (Craig Summers)]

I created an Index lifecycle policy to delete logs file entries older than 93 days to help with disk space and retention policies.

I noticed that I need to create an index template to link to the policy, so I created the one below:

However, when I try to link the retention policy to this index template, it doesn't show up in the list:

[wylie (Wylie Conlon)]

Looks like you are only seeing "legacy" index templates, which was fixed in the release of 7.10.0 a few hours ago. So you can either upgrade to use the Kibana UI, or you can work around the UI bug in older versions by running the API request directly. You can update the template settings manually by assigning the index.lifecycle.name to93-Day-Retention-Policy. There's also a way to assign this manually to indices even if they weren't matched by the template:

Configure a lifecycle policy | Elasticsearch Reference [7.10] | Elastic

Hopefully this helps!

[wylie (Wylie Conlon)]

One more option for you: you can enter this manually as JSON by going to the Edit Template UI, typing the same JSON key for index.lifecycle.name

[Craig2188 (Craig Summers)]

Would I have to update all components, so Elastic, Kibana, Logstash, Filebeat, Winlogbeat? Or could I get away with updating just Kibana?

I haven't used the JSON stuff before, how would I apply it via this method?

[Craig2188 (Craig Summers)]

So, I created the following:

POST logstash-*/_delete_by_query
{
 "query": {
   "range": {
     "@timestamp": {
       "lte": "now-93d"
      }
    }
  }
}

POST winlogbeat-*/_delete_by_query
{
 "query": {
   "range": {
     "@timestamp": {
       "lte": "now-93d"
      }
    }
  }
}

POST filebeat-*/_delete_by_query
{
 "query": {
   "range": {
     "@timestamp": {
       "lte": "now-93d"
      }
    }
  }
}

Does this need to be manually run, or can it run automatically every so often somehow?
I presume this would be correct to delete anything older than 93 days for those queries? I ran them and they came back with successful results, although nothing was deleted as it hasn't been running for 93 days yet.

[wylie (Wylie Conlon)]

The purpose of Index Lifecycle Management is to automate that- you shouldn't need to manually delete documents. Like I said, you can do this from the Kibana UI by going to the "Edit template" UI. You can find it in:

Stack Management > Index Management > Index Templates > metricbeat-* > Edit > Page 3, Index settings

And then you can use the setting { "index.lifecycle.name": "93-Day-Retention-Policy" }