Processors drop_event - no worky

Elastic Stack Beats
1

running metricbeat on centos7
package name = metricbeat-5.1.1-1.x86_64

The documentation says to use processors:
https://www.elastic.co/guide/en/beats/metricbeat/current/configuration-processors.html#conditions

If I use processors in my metricbeat.yml the drop event does not work, it just ignores it and I get no error or anything in debug mode, but if I use filters instead of processors it works.

THIS DOES NOT WORK

      processors:
        - drop_event:
            when:
             regexp:
               mount_point: '^/(run|sys|proc|dev)'

THIS DOES WORK

      filters:
        - drop_event:
            when:
             regexp:
               mount_point: '^/(run|sys|proc|dev)'

and in my /var/log/metricbeat file i see this when using filters: instead of processors:

2017-01-09T16:34:44Z DBG  Processors: drop_event, condition=regexp: map[mount_point:^/(run|sys|proc|dev)]
2

Can you share your full config file?

3

THIS WORKS

#==========================  Modules configuration ============================
metricbeat.modules:
#------------------------------- System Module -------------------------------
- module: system
  metricsets:
    # CPU stats
    - cpu
    # System Load stats
    - load
    # Per filesystem stats
    - filesystem
    # Per process stats
    - process
  enabled: true
  period: 30s
  processes: ['.*']
  filters:
    - drop_event:
        when:
          or:
            - regexp:
                mount_point: '^/(run|sys|proc|dev)'
            - regexp:
                device_name: '^10.69.69'
            - equals:
                device_name: sunrpc
#================================ General =====================================
name: metricbeat
#================================ Outputs =====================================
#-------------------------- logstash output ------------------------------
output.logstash:
  hosts: ["logstash:5045"]
  ssl.certificate_authorities: ["/etc/ssl/ca-chain.crt"]
  ssl.certificate: "/etc/ssl/client_cert.crt"
  ssl.key: "/etc/ssl/client_key.key"
  ssl.supported_protocols: ["TLSv1.2"]

THIS DOES NOT WORK

#==========================  Modules configuration ============================
metricbeat.modules:
#------------------------------- System Module -------------------------------
- module: system
  metricsets:
    # CPU stats
    - cpu
    # System Load stats
    - load
    # Per filesystem stats
    - filesystem
    # Per process stats
    - process
  enabled: true
  period: 30s
  processes: ['.*']
  processors:
    - drop_event:
        when:
          or:
            - regexp:
                mount_point: '^/(run|sys|proc|dev)'
            - regexp:
                device_name: '^10.69.69'
            - equals:
                device_name: sunrpc
#================================ General =====================================
name: metricbeat
#================================ Outputs =====================================
#-------------------------- logstash output ------------------------------
output.logstash:
  hosts: ["logstash:5045"]
  ssl.certificate_authorities: ["/etc/ssl/ca-chain.crt"]
  ssl.certificate: "/etc/ssl/client_cert.crt"
  ssl.key: "/etc/ssl/client_key.key"
  ssl.supported_protocols: ["TLSv1.2"]

The only thing different is using "filters:" instead of "processors:" makes it work

Also, in case any one asks.... configtest passes with processors in it

[root@web10 ~]# /usr/share/metricbeat/bin/metricbeat -c /etc/metricbeat/metricbeat.yml -configtest
Config OK
4

Thanks for sharing the full config. Inside the modules it is called filters, on a global level it is called processors. This is for historical reason. We already had some discussions in the past about this on how to resolves this but there was not final conclusion / fix yet. @dedemorton FYI

About configtest passing: We can't detect currently fields which are "too much" with config test.

5

maybe you guys should change the documentation

2 Likes
6

Ok, I figured more out with processors....
This config filtering works using "processors:"

#==========================  Modules configuration ============================
metricbeat.modules:
#------------------------------- System Module -------------------------------
- module: system
  metricsets:
    # CPU stats
    - cpu
    # System Load stats
    - load
    # Per filesystem stats
    - filesystem
    # Per process stats
    - process
  enabled: true
  period: 30s
  processes: ['.*']
#================================ Processors ==================================
processors:
  - drop_event:
      when:
        or: 
          - regexp:
              system.filesystem.mount_point: '^/(run|sys|proc|dev)'
          - regexp:
              system.filesystem.device_name: '^10.69.69'
          - equals:
              system.filesystem.device_name: sunrpc
          - and:
              - equals:
                  metricset.name: process
              - not:
                  regexp:
                    system.process.cmdline: '.*'
#================================ General =====================================
name: metricbeat
#================================ Outputs =====================================
#-------------------------- logstash output ------------------------------
output.logstash:
  hosts: ["logstash:5045"]
  ssl.certificate_authorities: ["/etc/ssl/ca-chain.crt"]
  ssl.certificate: "/etc/ssl/client_cert.crt"
  ssl.key: "/etc/ssl/client_key.key" 
  ssl.supported_protocols: ["TLSv1.2"]
7

Yes. Did it with filters local to modules not work?

8

using the last config enables more granular filtering because of the field names "system.filesystem"

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.