We are heavily starting to use filebeat (to push to kafka) and we have a configuration with 100+ prospectors (100+ kafka topics)
For each prospector we repeat some parameters like encoding, ignore_older, scan_frequency, ...
We think filebeat would benefit of either:
- a way to change the default value for all prospectors (for example the default value of max_bytes is currently 10MB (10485760). If I want to set 20MB for all my prospectors, I have to copy past the parameter for each of them. This works for people having many prospectors of the same kind
- a way to define templates: when creating a propsector, we can import values from a template or another. This works for people having many prospectors of a few kinds
- a way to copy parameters from a previously defined prospector
For now, we use puppet to create our configuration file so we use a loop in our ERB file and this works fine. The only problem is that the deployed configuration file is quite long and can be reduced with one of the options above