Apply Index Life Cycle Policies

#1

Hi All,

I have been trying to set up the index life cycle policies to my active index already available.
My index names are X, Y, Z. Trying to assign a policy to roll over once reaching 25 GB.
I'm creating the index name in the output part of logstash.
I went through multiple docs provided by elastic.co. But I feel it is not clear or straight forward.
The docs mentioned to create alias name for my index and it throws error that it does not match the pattern like X-000001, Y-000001, Z-000001.
I tried creating an alias of the same way but getting the error "index name does not match pattern '^.*-\d+$'"
Could some one share the steps correctly on the following

  • how the index name should be present in logstash.
  • how to create a template in Kibana to map to an index.
  • how to create an index which satisifes the roll over condition.
#2

This is the official guide for setting up ILM. Which steps have you already completed successfully?

The Logstash elasticsearch output plugin has additional configurations for ILM.

#3

Hi @nickpeihl,

Thanks for the correct link related to the elastic search output plugin for logstash.
As per the example in the doc
output {
elasticsearch {
ilm_rollover_alias => "custom"
ilm_pattern => "000001"
ilm_policy => "custom_policy"
}
}
once the roll over policy condition is met, will the index pattern gets incremented automatically to 000002, or do we need to set it in any specific regex pattern to get it incremented?

#4

According to the docs, the pattern automatically gets incremented when the indices rollover. https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-ilm_pattern

In order to work correctly your ilm_pattern needs a dash before the 000001 (ex. mypattern-000001. But you could also remove the ilm_pattern from the config and it will use a default pattern of {now/d}-000001.

#5

As per the docs, I have created the ilm_pattern and the indices are mapped to a policy when roll up should trigger when 15 GB reaches. One of my indices to which i have mapped the 15 GB policy has reached the storage size of 15 GB but the indices doesn't seem to roll up and a new index with pattern -000002 is not created.

Did I miss any configuration or a step for the roll over to trigger?

#6

The new index got created with the pattern 000002 after sometime i updated the comment here.