ILM setup dynamic alias in filebeat

[TanguyB]

Hello,

I'm trying to setup ILM in my filebeat.yml and I want to use dynamic index

My index is set to:

index: "filebeat-%{[agent.version]}-%{[fileset.module]}-%{+yyyy.MM.dd}"

which mean I'll have different index per filebeat module like :

• filebeat-7.1.1-apache-2019.06.25
• filebeat-7.1.1-audit-2019.06.25
• ...

My ILM conf is set to :

setup.ilm.enabled: auto
setup.ilm.rollover_alias: "filebeat-%{[agent.version]}-%{[fileset.module]}"
setup.ilm.pattern: "{now/d}-001"

But it doesn't work, it seems that we can't setup a dynamic alias for ILM

Doc mention to manually setup rollover index ( I don't know which values to set there..), does it mean I can't use the automatic way via filebeat ?
https://www.elastic.co/guide/en/beats/functionbeat/6.7/ilm.html#_advanced_ilm_settings

Thanks

[chandra2037 (Chandra)]

You are right, you cannot setup dynamic alias for ILM.

One option if you want to load ilm through filebeat - create static alias name by indexing all modules data into single index and add the dynamic value as an additional field (eg: fileset.module) in the index document, then you can apply the filter on the consumption side.

Otherwise, you can load the ilm policies and templates through Elastic API externally.
For loading externally you can follow the below approach

1. If you have a limited set of values for "fileset.module" values, then create a separate template for each one with separate alias

eg:

PUT _template/my_template {
"index_patterns": ["filebeat-7.1.1-apache-*"],
"settings": {
"number_of_shards": 1,
"number_of_replicas": 1,
"index.lifecycle.name": "filebeat-7.1.1-apache-policy",
"index.lifecycle.rollover_alias": "filebeat-7.1.1-apache"
}
}

filebeat-7.1.1-apache-policy - ILM Policy name

PUT _template/my_template {
"index_patterns": ["filebeat-7.1.1-audit-*"],
"settings": {
"number_of_shards": 1,
"number_of_replicas": 1,
"index.lifecycle.name": "filebeat-7.1.1-audit-policy",
"index.lifecycle.rollover_alias": "filebeat-7.1.1-audit"
}
}

Then bootstrap the process by creating first index

#PUT /<filebeat-7.1.1-apache-{now/d}-1> with URI encoding:

PUT /%3Cfilebeat-7.1.1-apache-%7Bnow%2Fd%7D-1%3E {
"aliases": {
"filebeat-7.1.1-apache":{
"is_write_index": true
} } }

Then in your filebeat conf..
index: "filebeat-7.1.1-%{[fileset.module]}"
Note: As I said assuming you setup templates/ilm policies for all combinations of "fileset.module" values.