Simplification of filebeat.yml?

Elastic Stack Beats
1

What would be the best way to configure Filebeat to ship multiple rolling log types while simplifying the configuration?

I have ~30 of these files to monitor and was hoping to use reference variables to streamline the configuration. Most of the file-specific settings will be the same while certain file-specific attributes will be unique to each file, if that makes any sense. Any input would be appreciated.

- input_type: log
  paths:
- /path/a.log
  fields:
    document_type: log
    type: log-a
  enabled: true
  ignore_older: 7h
  document_type: log
  scan_frequency: 1s
  clean_inactive: 8h
  close_timeout: 7h
  close_inactive: 7h
  close_renamed: true

- input_type: log
  paths:
- /path/b.log
  fields:
    document_type: log
    type: log-b
  enabled: true
  ignore_older: 7h
  document_type: log
  scan_frequency: 1s
  clean_inactive: 8h
  close_timeout: 7h
  close_inactive: 7h
  close_renamed: true
2

filebeat doesn't support global defaults, but it support referencing other settings via ${full-name}.

If you have common settings and might want to change them all at once you can do this:

filebeat.prospectors:
- input_type: log  # < I think this is the default and can be removed
  paths: ...
  fields:
    document_type: log
    type: log-b
  enabled: true # < always true by default, not required. You can use variables to have a class of prospectors to be enabled/disabled at once like:
  # enabled: ${defaults.class.logs}
  ignore_older: ${defaults.ignore_older}
  scan_frequency: ${defaults.scan_frequency}
  ...

defaults:
  ignore_older: 7h
  scan_frequency: 1s
  class:
    logs: true
    other: false

Using the class idea, you can have defaults by class like:

defaults:
  ignore_older: 7h
  scan_frequency: 1s
  class:
    logs.ignore_older: ${defaults.ignore_older}
    logs.scan_frequency: ${defaults.scan_frequency}
    app.ignore_older: 24h
    app.scan_frequency: ${defaults.scan_frequency}

See docs for filebeat config files features. E.g. you can use environment variables and defaults or start filebeat with -E <varname>=<value> to overwrite variables in the defaults namespace.

With this much repetition, you might also consider a script writing a configuration file before starting filebeat using templates.

1 Like
3

I just wrote a quick script to generate out the config. Unfortunate there's no simple way to do a more global override/default for templating of sub-values. Thanks!

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.