Are Beats designed to be used with ILM?

Elastic Stack Beats
, , ,
1

I'm trying to implement ILM to Filebeat indices, but this popped up in the process:

We have different Filebeat versions and multiple modules like docker, access, and many others, but for simplicity, let's take a look at docker only.

We use labels for each Filebeat module, so we can differentiate them in Logstash and separate them in different indices, for example in Logstash we do - if label == docker, index => [metadata-beat]-[metadata-version]-docker

In this way, we will have indices:
filebeat-6.8.0-docker
filebeat-7.6.0-docker
filebeat-7.11.1-docker

With this setup, having different index templates for each version is not a problem, since they match index pattern filebeat-version-*, but with ILM we don't have a way to add Filebeat version in the name dynamically.

Seems like the only way is to manually create rollover index for each filebeat version and module, but that will be a lot of work, since we have other beats with the same logic, and i will need to repeat the process for every new version that come.

Is there a way to achieve this in a more automated manner?

2

I know that there is the setup pattern property: Configure index lifecycle management | Filebeat Reference [7.13] | Elastic . Not sure if it helps you with your case.

3

Thanks, that will be useful! Is there a way to combine this features with option to output each module to different index dynamically?
Something like - setup.ilm.pattern: "%{[event][module]}-{now/d}-000001" (this one did not work)

4

I tried with
setup.ilm.rollover_alias: "metricbeat-%{[agent.version]}-%{[service.type]}"
as well, no success. It creates template at the start of the service, so maybe it's not desighned to be dynamic.

5

Correct. You'd have to either manually create the policies and templates and then u can dynamically send the data to the respective alias using the conditional settings on the elasticsearch output.