Elasticsearch Index Management - Index Lifecycle Policies

[brettlarsonwci (Brett Larson)]

Hello All -

I am currently trying to set up some lifecycle policy's to clean up indices. In Kibana, I have an index pattern of "logstash-*". Fluentd is just taking everything matching that pattern and sending it. This creates a new indice each day such as logstash.2020.08.01. I could apply the lifecycle policy directly to the indice, but I would have to go in each day and do that to each subsequent indice that gets automatically created. I assume the correct way to go about this is to create an index template, and apply the lifecycle policy to the template? And then somehow edit my fluentd configuration to direct it to a given index template instead of just a kibana index pattern? Am I on the right track? Also wondering what to put for "Alias" in this scenario, because when I attempt to bind a lifecycle policy to a template, it says with rollover enabled you have to specify alias. Not sure what is appropriate for that.

Thanks so much for your help!

[Nathan_Reese]

Sounds like you are trying to set up automatic rollover? Have you tried https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html?

[warkolm (Mark Walkom)]

@brettlarsonwci

I assume the correct way to go about this is to create an index template, and apply the lifecycle policy to the template

Yep, see Manage existing indices | Elasticsearch Guide [7.9] | Elastic

@brettlarsonwci

And then somehow edit my fluentd configuration to direct it to a given index template instead of just a kibana index pattern

Into an index, the template will apply (if you're using a matching name of course).

@brettlarsonwci

Am I on the right track?

Definitely