I'm trying to implement ILM to Filebeat indices, but this popped up in the process:
We have different Filebeat versions and multiple modules like docker, access, and many others, but for simplicity, let's take a look at docker only.
We use labels for each Filebeat module, so we can differentiate them in Logstash and separate them in different indices, for example in Logstash we do - if label == docker, index => [metadata-beat]-[metadata-version]-docker
In this way, we will have indices:
filebeat-6.8.0-docker
filebeat-7.6.0-docker
filebeat-7.11.1-docker
With this setup, having different index templates for each version is not a problem, since they match index pattern filebeat-version-*, but with ILM we don't have a way to add Filebeat version in the name dynamically.
Seems like the only way is to manually create rollover index for each filebeat version and module, but that will be a lot of work, since we have other beats with the same logic, and i will need to repeat the process for every new version that come.
Is there a way to achieve this in a more automated manner?
Thanks, that will be useful! Is there a way to combine this features with option to output each module to different index dynamically?
Something like - setup.ilm.pattern: "%{[event][module]}-{now/d}-000001" (this one did not work)
I tried with setup.ilm.rollover_alias: "metricbeat-%{[agent.version]}-%{[service.type]}"
as well, no success. It creates template at the start of the service, so maybe it's not desighned to be dynamic.
Correct. You'd have to either manually create the policies and templates and then u can dynamically send the data to the respective alias using the conditional settings on the elasticsearch output.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.