Filebeat Nginx module not dropping events

Elastic Stack Beats
1

I'm trying to exclude some events, started out with a more complex processor, but was never once able to make even a simple condition work. drop_event, with no condition, does what it's supposed to and drops everything. But as soon as a try to apply a condition, it refuses to drop any event at all. Here's the nginx.yml
module config file:

- module: nginx

  # Access logs
  access:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:
    input:
      processors:
      - drop_event:
          when:
            equals:
              nginx.access.remote_ip: '127.0.0.1'

which should drop any request coming from the system itself. It doesn't. I've also tried to match nginx.access.url and it fails to drop a single event involving the URL in the rule. It shouldn't be this hard to get it to do what it's supposed to do!

2

UPDATE: I wrote before the config should be on the module level. This is NOT correct and what you have above is correct.

Any chance you could enable debug logging and share the output. Currently trying this out locally.

3

Can you replace equals through contains in your processor above?

4

I think the problem above is that nginx.access.remote_ip is only available after the ingest processor. The processing of the log line happens in Elasticsearch, so filebeat cannot filter based on this field as it never sees it.

For your case you would either have to do this in the ingest pipeline or do a regexp on the filebeat side with exclude_line: https://www.elastic.co/guide/en/beats/filebeat/6.5/filebeat-input-log.html#filebeat-input-log-exclude-lines

5

Sorry, things got hectic around here for a bit. I was suspecting something like that was the case, as I'd used the drop_event processor before, but without using the IP address. My ideal case for this would use both nginx.access.url and remote_ip to decide when to drop the event (one example: drop all events for "/server-status" coming from from a list of IPs representing the various performance analyzers I use).

Is url something that only gets set by ingest, or would it be available to use?

EDIT: Upon further digging, this can work in the minx module, but as @ruflin notes, the nginx fields aren't available at the time of running, so the only way to do it is to concoct a regex test against "message", i.e.:

- drop_event.when.regexp.message: "127.0.0.1.*"

in order to drop all events coming from 127.0.0.1

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.