Help with Processors in filebeat modules

Elastic Stack Beats
1

Hi, I'm having a lot of issues trying to figure out how to filter out log lines before they are indexed. After failing using "exclude_lines" for a couple of times, I quickly moved to the use of processors.
The main constraint I have is that I am using Saltstack to apply the configuration and therefore I'm trying to use as few nested clauses as possible. Here's the code I'm using:

  • /etc/filebeat/modules.d/apache.yml
- module: apache
  access:
    enabled: True
    processors:
      - drop_event.when.and:
        - equals.http.response.code: 302
        - equals.source.ip: "172.21.205.252"
        - equals.http.request.method: "HEAD"
  • /etc/filebeat/modules.d/system.yml
- module: system
  syslog:
    enabled: True
    processors:
      - drop_event.when.contains.message: lxc-container-default-with-nfs
  auth:
    enabled: True

These are the messages I am trying to get rid of:

  • [apache][access] 172.21.205.252 - "HEAD / HTTP/1.0" 302 undefined

  • [7580482.483661] audit: type=1400 audit(1579509159.591:12505): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-with-nfs" name="/" pid=7990 comm="(imedated)" flags="rw, rslave

I believe the issue might be in taking too much advantage of the YAML format and joining all commands, but I need some reassurance and hopefully some other ideas.
Thanks in advance.

2

Hi @Jose_E and welcome to discuss! :slight_smile:

I think the problem is in how Filebeat processes the log lines. For most modules, including the apache one, the parsing is done in ingest pipelines. These pipelines are executed when the events are received in Elasticsearch. When processors are executed, the module-specific fields (like http.response.code) don't exist yet in the event.

If you want to drop messages using filebeat processors you need to do it based on the content of the raw log line, something like what you are doing with the messages containing lxc-container-default-with-nfs.

3

Hi @jsoriano thank you for your reply!

I understand the problem with the ingestpipelines, and makes plenty of sense droping events based on the raw log lines. The processor with [quote="Jose_E, post:1, topic:215711"]
lxc-container-default-with-nfs [/quote] didn't work either though. So based on what you just told me I assume something like:

  • /etc/filebeat/modules.d/apache.yml
- module: apache
  access:
    enabled: True
    processors:
      - drop_fields.fields:
        - "302"
        - "172.21.205.252"
        - "HEAD"
  • /etc/filebeat/modules.d/system.yml
- module: system
  syslog:
    enabled: True
    processors:
      - drop_fields.fields: 
        - "lxc-container-default-with-nfs"
  auth:
    enabled: True

Could potentially work?

1 Like
4

I was proposing something more like this:

- module: apache
  access:
    enabled: True
    processors:
      - drop_event.when.and:
        - contains.message: 302
        - contains.message: "172.21.205.252"
        - contains.message: "HEAD"

But I wonder why the configuration you had for lxc-container-default-with-nfs was not working :thinking:

5

Hello, I'm new with filebeat and I'm in trouble adding a processor to the haproxy module.

To begin, just adding a tag would be enough, I tried with this config without much luck (Filebeat 7.1.1 on Debian stretch):

/etc/filebeat/modules.d/haproxy.yml

- module: haproxy
  # All logs
  log:
    enabled: true
    # Set which input to use between syslog (default) or file.
    #var.input:
    var.input: "file"

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/log/haproxy.log"]

    processors:
      - add_tags:
          tags: [test]

The processor is NOT working. If sat globally on /etc/filebeat/filebeat.yml it works.
Any help would be very appreciated. Many thanks.

6

Hey @aventrax, could you please create a new topic for your question?

7

Hey @Jose_E,

One thing after thinking twice on where processors can be defined after reading this topic: How to use processors in filebeat HAproxy's module?

Processors must be defined at the global config or at the input config, so when using modules you may need to define it like this:

- module: apache
  access:
    enabled: True
    input:
      processors:
        - drop_fields.fields:
          - "302"
          - "172.21.205.252"
          - "HEAD"
8

Hey @jsoriano, thanks for your reply it gave me the solution. The topic you found was pretty useful as well :smile:

So this is how I left the code and worked wonders:

- module: system
  syslog:
    enabled: true
    input.processors:
      - drop_event.when.contains.message: "lxc-container-default-with-nfs"

- module: apache 
  access:
    enabled: true
    input.processors:
      - drop_event.when.and:
        - contains.message: "302"
        - contains.message: "172.21.205.252"
        - contains.message: "HEAD"
9

Hey there, sorry to bring this back up. But apparently ever since I made these changes, filebeat simply stoped sending module events... I just found out about it. I'm sure it is related to the processors thingy, any idea why?

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.