I'm trying to implement ILM to Filebeat indices, but this popped up in the process:
We have different Filebeat versions and multiple modules like docker, access, and many others, but for simplicity, let's take a look at docker only.
We use labels for each Filebeat module, so we can differentiate them in Logstash and separate them in different indices, for example in Logstash we do - if label == docker, index => [metadata-beat]-[metadata-version]-docker
In this way, we will have indices:
With this setup, having different index templates for each version is not a problem, since they match index pattern filebeat-version-*, but with ILM we don't have a way to add Filebeat version in the name dynamically.
Seems like the only way is to manually create rollover index for each filebeat version and module, but that will be a lot of work, since we have other beats with the same logic, and i will need to repeat the process for every new version that come.
Is there a way to achieve this in a more automated manner?