[Index Lifecycle Management] Dynamic rollover alias and template name

Elastic Stack Logstash
1

Use Case:

Could it be possible to allow using variables in the properties "ilm_rollover_alias" as well as "template_name" in the Elasticsearch output, like we can do with the "index" property?
The configuration below raises an error on parsing the value of ilm_rollover_alias and template_name:

output {
if [@metadata][index] {
elasticsearch {
hosts => ["{ELASTICSEARCH_HOST}:{ELASTICSEARCH_PORT}"]
ilm_enabled => "true"
ilm_policy => "default-policy"
ilm_rollover_alias => "%{[@metadata][index]}"
template_name => "%{[@metadata][index]}"
}
}
}

While this is not working I use an IF statement for each different indices I need:

output {
if [@metadata][index] == "nginx" {
elasticsearch {
hosts => ["{ELASTICSEARCH_HOST}:{ELASTICSEARCH_PORT}"]
ilm_enabled => "true"
ilm_policy => "default-policy"
ilm_rollover_alias => "nginx"
template_name => "nginx"
}
}
if [@metadata][index] == "app-logs" {
elasticsearch {
hosts => ["{ELASTICSEARCH_HOST}:{ELASTICSEARCH_PORT}"]
ilm_enabled => "true"
ilm_policy => "default-policy"
ilm_rollover_alias => "app-logs"
template_name => "app-logs"
}
}
}

2

Hi @Cyril_Berber

The template is used only at pipeline startup time, where the output plugin will install the template in template_path if manage_template is set to true (default).
So in this context it's not possible to accept a template name per event, as these management tasks should be either before indexing data.

In the case of ilm_rollover_alias the problem is similar: the ilm_* options are used to install an ILM policy at plugin startup time so it does not make sense here to accept values from events to configure the policy.

If you're managing ILM policies outside of logstash (see the documentation here) then you just can set ilm_enabled => false and just use the normal index option which supports fetching the value per event.

1 Like
3

If ILM policies are managed outside of logstash, does the first index need to be created explicitly?

In our setup, we have different indices for, for example, Kubernetes namespaces: logstash-kube_<ns>-<date>. Different indices have different lifecycles, currently implemented using curator. Many of them also have their own ES index templates.

So if we were to move to ILM, and change to a logstash-kube_<ns>-<sequence> type index naming, would we need to always make sure that the logstash-kube_<ns>-00001 index is created beforehand and aliased as logstash-kube_<ns>, and then have logstash write to the alias?

Currently, adding new namespaces requires no manual actions in Logstash/ES and I'd really like to keep it that way. It would be great if Logstash could be configured not to manage the ILM policy itself, but still create the alias automatically for new indices.

4

But the rollover alias is not set on the policy, it's set on the index (same as ilm_policy).

I've got custom policies and custom templates (not in a 1-1 relationship), and really just want to switch from index => '%{type}-%{+YYYY.MM.dd}' to ilm_rollover_alias => '%{type}'

If you're managing ILM policies outside of logstash then you just can set ilm_enabled => false

That's not what the documentation says - it says to use ilm_policy.

1 Like
5

The ilm_policy should probably also be dynamic, and set directly on the index (especially if manage_template => false).

6

It would be great if Logstash could be configured not to manage the ILM policy itself

The documentation says that's what ilm_policy is for:

Modify this setting to use a custom Index Lifecycle Management policy, rather than the default. If this value is not set, the default policy will be automatically installed into Elasticsearch

7

This cannot be done dynamically, as the initial index and alias are set at Logstash init time, if they haven't been set yet.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

9

This is becoming a common ask in the field.
For those interested in an enhancement request to make this type of setup easier, please +1 to the issue here.