Dynamic rollover alias and template name

Elastic Stack Elasticsearch
1

Hi.

Could it be possible to allow using variables in the properties " ilm_rollover_alias " as well as " template_name " in the Elasticsearch output, like we can do with the "index" property?

Index lifecycle error
illegal_argument_exception: index.lifecycle.rollover_alias [server-log-*-] does not point to index [server-log-demoproduct-2021.08.03]
Stack trace

My indices is "server-log-%{product}-%{+YYYY.MM.dd}"

2

In the output of what sorry?

3

Hi. This is error in kibana
java.lang.IllegalArgumentException: index.lifecycle.rollover_alias [server-log-*-] does not point to index [server-log-demovn-2021.08.03]
at org.elasticsearch.xpack.core.ilm.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:124)
at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:175)
at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:335)
at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggered(IndexLifecycleService.java:273)
at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:184)
at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:217)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
at java.base/java.lang.Thread.run(Thread.java:831)

My template setting
{
"index": {
"lifecycle": {
"name": "server-log-policy",
"rollover_alias": "server-log-*-"
},
"refresh_interval": "5s",
"number_of_shards": "1",
"number_of_replicas": "0"
}
}

logstash output

elasticsearch {
hosts => ["http://10.10.10.8:9200"]
#ilm_enabled => "true"
#ilm_pattern => "000001"
#ilm_policy => "server-log-policy"
#ilm_rollover_alias => "server-log"
#manage_template => false
user => "elastic"
password => "xxx"
index => "server-log-%{product}-%{+YYYY.MM.dd}"
}

4

Ok that makes more sense, you mean Logstash not Elasticsearch :slight_smile:
When you send data from Logstash to an ILM policy in Elasticsearch, you need to send it to the write alias and not the underlying index pattern.

Just a quick ask for the future - please format your code/logs/config using the </> button, or markdown style back ticks. It helps to make things easy to read which helps us help you :slight_smile:

5

Thank for your reply. I have changed configure in logstash to send data to the write alias.

elasticsearch {
                        hosts => ["http://10.10.10.8:9200"]
                        ilm_enabled => "true"
                        #ilm_pattern => "000001"
                        ilm_policy => "server-log-policy"
                        ilm_rollover_alias => "server-log-%{product}"
                        #manage_template => false
                        user => "elastic"
                        password => "1DoQN06dmFOIdU3ZZhPJ"
                        #index => "server-log-%{gameid}-%{+YYYY.MM.dd}"
                }

But logstash is failed

[2021-08-04T08:12:33,255][ERROR][logstash.outputs.elasticsearch][log] Failed to install template. {:message=>"Malformed escape pair at index 23: /_template/server-log-%{product}"

I found a similar problem here Impossible to use variables in ilm_rollover_alias · Issue #10962 · elastic/logstash · GitHub

6

Correct, you cannot use variables there unfortunately.

7

Thank for your reply.
I will disable rollover option in ilm policy And manage rollover by the underlying index pattern in logstash configuration.