How to avoid logstash elasticsearch output ilm_rollover_alias overwriting the index prefix

[Louis_Zhang]

I expecting to get index mobile-2020.05.06-000001 with alias 'streamdata-alias', but the logstash creates index streamdata-alias-2020.05.06-000001, is the a bug?

Alias and index name are 2 tings, why they have to be same in name, in the prefix?

output   {
    elasticsearch {
      ilm_rollover_alias => "streamdata-alias"
      ilm_policy => "streamdata-policy"
      index => "mobile-{now/d}-000001"
    }
}

I think it should like this,
if the index exists, it wirte the event into the alias with the is_write_index = true.
if index not exists, create the index with the alias and set the is_write_index = true and write event to the alias.

[Badger]

I do not believe the index option is used if you are using ILM.

[Louis_Zhang]

yeah, according to the code it use the alias direclty. logstash-output-elasticsearch, line95, not sure if it's possible to change to

def rollover_alias_target
      "<#{@index}-#{ilm_pattern}>"
 end

[Louis_Zhang]

hmm, found something interesting,
haven't tried, if I am correct, when I set the alias with the date, it'll write the event into a new index without ilm alias the next day, if logstash not reboot, beacuse the 'setup_ilm' called only one time when it's launched.

elasticsearch {
...
    ilm_rollover_alias => "alias-{now/d}"
    ilm_pattern => "000001"
...
}

[Luca_Belluccini]

Hello @Louis_Zhang

Logstash bootstraps the write alias at startup.

Given the current design of the ILM APIs, it is not possible to use variable substitution in ilm_rollover_alias or ilm_pattern.

When ILM is enabled and ILM settings are used on Logstash, the index parameter is not used.

It is not possible to have totally different index names and alias name when using the Logstash ILM settings: the actual index will be named ilm_rollover_alias + ilm_pattern.

Do not add the date math pattern "{now/d}" in the ilm_rollover_alias. Set it to the ilm_pattern: the alias must stay the same as Logstash cannot and doesn't need to know when the index rollovers.
It's Elasticsearch responsibility with the ILM policy to rollover the index depending on the size, age or doc count.

See more at our documentation.

Please let me know if it is clear or if you have issues to set it up.
To be sure everything is correct, stop Logstash, change the parameters, verify there is no index template automatically installed already by Logstash and then restart Logstash.
Doing do, at first start, an index template will be installed together with a default policy.
The rollover alias will be setup and associated to an actual index (e.g. myindex and it will first point to myindex-2020...-00001).
From now on Logstash will continue to write to myindex and Elasticsearch will check every 10 minutes (default) if the rollover must take place depending on the ILM policy of the index.