System module: error when defining syslog.input.processors

[TatsuKishi]

Hey, I'm trying to define a processor in the system module just for syslog to drop certain spammy, non-relevant events. But when I define a drop_event I get an error about a processor having multiple actions:
#011ERROR#011fileset/factory.go:105#011Error creating input: each processor must have exactly one action, but found 2 actions (add_locale,drop_event)

According to the documentation that should be a valid location: https://www.elastic.co/guide/en/beats/filebeat/current/defining-processors.html#where-valid
I also tried defining it in the modules top scope which failed with "Fileset system/input is configured but doesn't exist"

I'd rather avoid defining the processor at top scope since I have other inputs where this filter would just cause unnecessary computation.

Any help for this would be appreciated.

Config

- module: system
  # Syslog
  syslog:
    enabled: true
    input:
      processors:
      - drop_event:
          when:
            and:
            - contains:
                message: "error on subcontainer"
            - contains:
                message: "ia_addr"

  # Authorization logs
  auth:
    enabled: true

[jsoriano (Jaime Soriano)]

Hey @TatsuKishi, welcome to discuss

This error can be a side effect of having processors predefined in modules. This add_locale processor is used by default, but your config tries to override it. Could you try to include this processor in your config? It would be something like this:

- module: system
  # Syslog
  syslog:
    enabled: true
    input:
      processors:
      - add_locale: ~
      - drop_event:
          when:
            and:
            - contains:
                message: "error on subcontainer"
            - contains:
                message: "ia_addr"

  # Authorization logs
  auth:
    enabled: true

Another option you have is to define your drop_event processor in the main configuration file of filebeat, but then you have to take into account that the processor would be applied to all events generated.

[TatsuKishi]

Adding the "add_locale" to the processors does the job, thank you!

Another thing I noticed is that this error was not picked up when doing a "filebeat check config". That threw me for quite the loop since it went through our CI due to that when it really shouldn't have.

[jsoriano (Jaime Soriano)]

@TatsuKishi

Another thing I noticed is that this error was not picked up when doing a "filebeat check config". That threw me for quite the loop since it went through our CI due to that when it really shouldn't have.

Yeah, probably filebeat considers this configuration correct, so config checks work, but later it cannot instantiate the pipelines because of the conflict with the predefined processors.