Using processors with modules in Filebeat

Elastic Stack Beats
1

TL;DR

How do I add fields (or any processors) to the config for a preexisting module without editing the module source?

Issue

I'm attempting to add some fields to logs ingested via the system module. This is my modules.d/system.yml:

# Module: system
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.9/filebeat-module-system.html

- module: system
  # Syslog
  syslog:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

    # Add additional required fields.
    processors:
      - add_fields:
          target: ''
          fields:
            application:
              name: 'myapp'
              env: 'dev'

  # Authorization logs
  auth:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

    # Add additional required fields.
    processors:
      - add_fields:
          target: ''
          fields:
            application:
              name: 'myapp'
              env: 'dev'

This loads correctly, but the additional fields I add aren't available. When I look at (for example) module/system/syslog/config/syslog.yml, I see:

type: log
paths:
{{ range $i, $path := .paths }}
 - {{$path}}
{{ end }}
exclude_files: [".gz$"]
multiline:
  pattern: "^\\s"
  match: after
processors:
  - add_locale: ~
  - add_fields:
      target: ''
      fields:
        ecs.version: 1.5.0

which makes me question whether this is possible, without editing that file, which isn't desirable, since it gets overwritten each time I update the filebeat, whereas modules.d/system.yml does not.

So my questions are...is it possible to add fields (or any processors) to the config for a preexisting module without editing the module source, and how?

1 Like
2

I am having a very similar problem - Specifically that the fingerprint processor doesn't seem to be running.

Here's my modules.d/netflow.yml

# Module: netflow
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.9/filebeat-module-netflow.html

- module: netflow
  log:
    enabled: true
    var:
      netflow_host: 0.0.0.0
      netflow_port: 2055
      processors:
        - fingerprint:
            fields: ["netflow.destination_ipv4_address","netflow.destination_transport_port","netflow.exporter.timestamp","netflow.source_ipv4_address","netflow.source_transport_port"]
            target_field: "@metadata._id"

I've done a little digging, and it seems that there is something of a disconnect as to how the modules parses/processes the inputs vs. how the type:netflow (in my case) parses

3

Try this - I had to put an input stanza on which the processor worked

# Module: netflow
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.9/filebeat-module-netflow.html

- module: netflow
  log:
    enabled: true
    var:
      netflow_host: 0.0.0.0
      netflow_port: 2055
    input:
      processors:
        - fingerprint:
            fields: ["netflow.destination_ipv4_address","netflow.destination_transport_port","netflow.exporter.timestamp","netflow.source_ipv4_address","netflow.source_transport_port"]
            target_field: "@metadata._id"
4

I'll try this - thanks. I've noticed that modules already have some processors specified (at least the system module, haven't looked at netflow). Have you noticed - does doing this override those processors, or does that data still appear?

5

I haven't looked at the system module much yet - I will do so shortly.

I'll be honest, I just started my elastic journey last week, and have been beating my head against some of the documentation. I hope that my struggles can help some other folks.

Frank

6

I'll be looking a the system module here shortly, will keep you updated. Welcome to the Elastic fellowship!

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.