No way to specify filebeat 8.0.0 data stream and index template pattern

[rouke-broersma]

In filebeat 7.x we were able to specify custom index template and rollover_alias for the indexes used by filebeat. We use this to split which indices different filebeats send their logs to because some filebeats collect general logfiles from different applications and some filebeats collect logs from specific products like for example nginx. We don't want this generic logging to be bundled with nginx logging so we don't use the same indices. After testing the upgrade to 8.0.0 we noticed that this setting was no longer honored and that there does not seem to be any alternative in 8.0.0 data streams. See: Configure index lifecycle management | Filebeat Reference [7.17] | Elastic

We cannot use Change the index name | Filebeat Reference [8.0] | Elastic because we use ILM.

How are we supposed to do this now? Or are we actually just trying to do something we should not be doing?

[kvch (Noémi Ványi)]

rollover_alias has been removed because it does not make sense in data streams. A data stream is basically an alias with backing indices, and the data stream name is the only alias you will ever need. You can either load the default ILM policy of Beats to manage the backing indices of the data stream. But we encourage you to use the UI in Kibana because it can lead you through the whole process: Tutorial: Automate rollover with ILM | Elasticsearch Guide [8.0] | Elastic

If I understood your use case correctly, you can just configure different setup.template.name and setup.template.pattern for each Filebeat instance you are running.

[rouke-broersma]

Thank, we will try using the template name and pattern as suggested.

[leandrojmp (Leandro Pereira)]

You can also use a custom template and set your ILM policy there.