Applying configuration to all prospectors

[vkurup (Vinod Kurup)]

Is it possible to apply configuration to 'all' prospectors? For example, I know i can set up 2 prospectors with multiline configuration like this:

filebeat.prospectors:
  - input_type: log
    paths:
      - /var/log/syslog
    multiline.pattern: '^[[:space:]]'
    multiline.negate: false
    multiline.match: after
    fields:
      type: syslog
  - input_type: log
    paths:
      - /var/log/rabbitmq/rabbit@*.log
    multiline.pattern: '^[[:space:]]'
    multiline.negate: false
    multiline.match: after
    fields:
      type: rabbitmq

Is it possible to write all those multiline items in just one place? I didn't see a clear example in the docs discussing that possibility.

[andrewkroh]

There is no built in way to do this, but if you are comfortable with YAML you can use YAML anchors. If you aren't comfortable with anchors and how they work then I would leave the config as you have it. For example:

macros:
  syslog_defaults: &syslog
    input_type: log
    multiline.pattern: '^[[:space:]]'
    multiline.negate: false
    multiline.match: after
    fields:
      type: syslog
  
filebeat.prospectors:
- <<: *syslog
  paths:
    - /var/log/rabbitmq/rabbit@*.log

- <<: *syslog
  paths:
    - /var/log/syslog
  fields:
    type: rabbitmq

The final config would look like this. (tested with ./filebeat -e -d "config")

{
  "filebeat": {
    "prospectors": [
      {
        "fields": {
          "type": "syslog"
        },
        "input_type": "log",
        "multiline": {
          "match": "after",
          "negate": false,
          "pattern": "^[[:space:]]"
        },
        "paths": [
          "/var/log/rabbitmq/rabbit@*.log"
        ]
      },
      {
        "fields": {
          "type": "rabbitmq"
        },
        "input_type": "log",
        "multiline": {
          "match": "after",
          "negate": false,
          "pattern": "^[[:space:]]"
        },
        "paths": [
          "/var/log/syslog"
        ]
      }
    ]
  }
}

[vkurup (Vinod Kurup)]

Thank you! I'll stick with my version, but nice to know this is possible.

[steffens (Steffen Siering)]

You can also reference other settings using the ${<setting-name>} syntax from any setting. Including full namespaces. e.g.

filebeat.prospectors:
  - input_type: log
    paths:
      - /var/log/syslog
    multiline: ${common.multiline}
    fields.type: syslog
  - input_type: log
    paths:
      - /var/log/rabbitmq/rabbit@*.log
    multiline: ${common.multiline}
    fields.type: rabbitmq

common.multiline:
  pattern: '^[[:space:]]'
  negate: false
  match: after

[andrewkroh]

That's certainly more friendly that anchors. Thanks @steffens.