[suggestion] Filebeat prospector configuration template

Elastic Stack Beats
1

We are heavily starting to use filebeat (to push to kafka) and we have a configuration with 100+ prospectors (100+ kafka topics)
For each prospector we repeat some parameters like encoding, ignore_older, scan_frequency, ...

We think filebeat would benefit of either:

  • a way to change the default value for all prospectors (for example the default value of max_bytes is currently 10MB (10485760). If I want to set 20MB for all my prospectors, I have to copy past the parameter for each of them. This works for people having many prospectors of the same kind
  • a way to define templates: when creating a propsector, we can import values from a template or another. This works for people having many prospectors of a few kinds
  • a way to copy parameters from a previously defined prospector

For now, we use puppet to create our configuration file so we use a loop in our ERB file and this works fine. The only problem is that the deployed configuration file is quite long and can be reduced with one of the options above

2

Good timing. We had a very similar discussion just recently internally. One question from my side is: As you auto generate the config file anyways, is the size of the config file an issue?

Probably the best here is to open a feature request on https://github.com/elastic/beats for further discussions.

3

No, the generated file size is not really an issue. At least not for filebeat. For OPs reading it to check its content, it can be.

And about opening a feature request on github, it is was I wanted to do first but ended up posting here because it says:

Please post all questions and issues on Beats - Discuss the Elastic Stack
before opening a Github Issue. Your questions will reach a wider audience there,
and if we confirm that there is a bug, then you can open a new issue.

4

@fld I really appreciate that you posted it here first and this is the recommended way. Like this we can check in advance if there is already something similar planned (or not). I this case I think it is definitively worth opening a Github issue.

5

Thanks, I just did

6

This topic was automatically closed after 21 days. New replies are no longer allowed.