Cannot drop_event by system.syslog.program field

[arustleund (Andrew Rustleund)]

Version: filebeat-6.6.2-darwin-x86_64

I am attempting to drop events from Filebeat using a processor. I have the system module enabled and I would like to drop events using conditions matching fields from the system module, but I haven't been able to get it to work yet. I was wondering if anyone had an idea of what I'm doing wrong. Here is the processors section of my filebeat.yml file:

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - drop_event:
      when:
        not:
          contains:
            system.syslog.program: "diagnosticd"

This seems to filter all events, even though some contain the program name "diagnosticd". The only way I've gotten drop_event to work is by using the field name "message", which seems to filter by the system.syslog.message field. However, using "program" as the field name does not seem to work either.

Any help would be greatly appreciated.

[steffens (Steffen Siering)]

Filebeat modules do not parse the contents in filebeat, but via Ingest Node in Elasticsearch. The raw events have no system.syslog fields yet, but only a message field with the original raw log line. Either you filter on message as you already do, or you adapt the ingest node pipeline in module/system/syslog/ingest/ to drop an event. In the later case the event will still be send, but dropped later, though.