Drop_event processor with not and or condition filters out all event

Elastic Stack Beats
1

Hello

I have configured a filestream with a drop_event processor. I want to drop the all event, which is not collected by Filebeat on hosts, which are listed in the configuration. The problem is the drop_event processors (version 7.17.6) drops all events despite a host is in the list.

The filstream configuration of the drop_event processor:

      - drop_event:
          when:
            not:
              or:
                - equals:
                    host.name: "dkrina01"
                - equals:
                    host.name: "dkrina03"
                - equals:
                    host.name: "dkrina04"
                - equals:
                    host.name: "dkrina05"
                - equals:
                    host.name: "dkrina993"
                - equals:
                    host.name: "dkrina996"
                - equals:
                    host.name: "eessi-rina04p"
                - equals:
                    host.name: "eessi-rina03p"
                - equals:
                    host.name: "eessi-rina04t"
                - equals:
                    host.name: "eessi-rina03t"
                - equals:
                    host.name: "eessi-gw-rina4a"

The filebeat documentation states the equals condition accepts only integer and string values.

When the processor is loaded the log reads:

2024-07-15T23:43:00.735+0200    DEBUG   [processors]    processors/processor.go:120     Generated new processors: add_fields={"event":{"dataset":"cas_audit.log"}}, drop_event, condition=!equals: map[host.name:0x5fa6203097e0] or equals: map[host.name:0x5fa6203097e0] or equals: map[host.name:0x5fa6203097e0] or equals: map[host.name:0x5fa6203097e0] or equals: map[host.name:0x5fa6203097e0] or equals: map[host.name:0x5fa6203097e0] or equals: map[host.name:0x5fa6203097e0] or equals: map[host.name:0x5fa6203097e0] or equals: map[host.name:0x5fa6203097e0] or equals: map[host.name:0x5fa6203097e0] or equals: map[host.name:0x5fa6203097e0]

I assume the host.name value is a pointer to the correct string values.

When the processor kicks in the filebeat log on f.x host dkrina9996 reads

2024-07-15T23:58:56.926+0200    DEBUG   [publisher]     pipeline/client.go:231  Pipeline client receives callback 'onFilteredOut' for event: {Timestamp:2024-07-15 23:58:56.92580185 +0200 CEST m=+966.711715891 Meta:null Fields:{"event":{"dataset":"cas_audit.log"},"input":{"type":"filestream"},"log":{"file":{"path":"/data/dkinst996/eessi/share/logs/cas/cas_audit.log"},"flags":["multiline"],"offset":3492231},"message":"2024-07-15T23:58:54,251 INFO [org.apereo.inspektr.audit.support.Slf4jLoggingAuditTrailManager] - Audit trail record BEGIN\n=============================================================\nWHO: audit:unknown\nWHAT: {source=RankedMultifactorAuthenticationProviderWebflowEventResolver, event=success, timestamp=Mon Jul 15 23:58:54 GMT+02:00 2024}\nACTION: AUTHENTICATION_EVENT_TRIGGERED\nAPPLICATION: CAS\nWHEN: Mon Jul 15 23:58:54 GMT+02:00 2024\nCLIENT IP ADDRESS: 127.0.0.1\nSERVER IP ADDRESS: 127.0.0.1\n=============================================================\n","tags":["e2e","cas_audit.log"]} Private:0xc000e981c0 TimeSeries:false}

Thanks, if you could point me to a solotion.

BR Flemming

2

Is this a mix of not and or that is always true? If host is dkrina01 is is not dkrina03, so dropped.

3

@rugenl According to the Filebeat documentation and other posts here, the processor should drop all events if a host.name is not in the list. That's logical or on the list members.

4

Hi,

you should change the not condition to and condition. Here's how you can modify your configuration:

- drop_event:
    when:
      and:
        - not.equals:
            host.name: "dkrina01"
        - not.equals:
            host.name: "dkrina03"
        - not.equals:
            host.name: "dkrina04"
        - not.equals:
            host.name: "dkrina05"
        - not.equals:
            host.name: "dkrina993"
        - not.equals:
            host.name: "dkrina996"
        - not.equals:
            host.name: "eessi-rina04p"
        - not.equals:
            host.name: "eessi-rina03p"
        - not.equals:
            host.name: "eessi-rina04t"
        - not.equals:
            host.name: "eessi-rina03t"
        - not.equals:
            host.name: "eessi-gw-rina4a"

This configuration will drop all events that are not from any of the listed hosts. Please try this and let me know if it works for you.

Regards

5

Hello @yago82

Thanks for your suggestion. However it does not work neither as required by my use case:

The problem with the and + not.equals configuration, is that it also drop events for the hosts, which are in the list. The Filebeat log reads, f.eks.:

2024-07-16T15:02:00.902+0200    DEBUG   [publisher]     pipeline/client.go:231  Pipeline client receives callback 'onFilteredOut' for event: