Can't drop events from metricbeat


(Robin Guo) #1

Hi folks,

We encountered an issue while the metricbeat is trying to send their metrics to elasticsearch.
even with some drop events rules, but can't drop the events we don't want. we still can see some particular values of a field.

metricbeat.yml

 - drop_event:
      when:
         or:
           - equals: 
                system.filesystem.device_name: "tmpfs"
           - equals: 
                system.filesystem.device_name: "devpts"
           - equals: 
                system.filesystem.device_name: "none"
           - equals: 
                system.filesystem.device_name: "sysfs"
           - equals: 
                system.filesystem.device_name: "proc"
           - equals: 
                system.filesystem.device_name: "cgroup"
           - equals: 
                system.filesystem.device_name: "hugetlbfs"
           - equals: 
                system.filesystem.device_name: "systemd-1"
           - equals: 
                system.filesystem.device_name: "devtmpfs"
           - equals: 
                system.filesystem.device_name: "debugfs"
           - equals: 
                system.filesystem.mount_point: "dev/shm"
           - equals: 
                system.filesystem.mount_point: "/dev/pts"
           - equals: 
                system.filesystem.mount_point: "/sys/fs/selinux"
           - equals: 
                system.filesystem.mount_point: "/proc"
           - equals: 
                system.filesystem.mount_point: "/proc/*"
           - equals:
                sytem.filesystem.mount_point: "/dev"
           - equals:  
                sytem.filesystem.mount_point: "/dev/hugepages"
           - equals:  
                sytem.filesystem.mount_point: "/dev/mqueue"
           - equals:  
                sytem.filesystem.mount_point: "/proc/sys/fs/binfmt_misc"
           - equals: 
                sytem.filesystem.mount_point: "/sys/fs/cgroup/blkio"
           - equals:  
                sytem.filesystem.mount_point: "/sys/fs/cgroup/cpu,cpuacct"
           - equals:  
                sytem.filesystem.mount_point: "/sys/fs/cgroup/cpuset"
           - equals:  
               sytem.filesystem.mount_point: "/sys/fs/cgroup/devices"

The results from grafana,but also can get the same results from kibana

it would be great if there is a way can drop some events start with /sys/*/.... with a wildcard.

Thanks


(Carlos Pérez Aradros) #2

It's possible, I think you want to check https://www.elastic.co/guide/en/beats/metricbeat/current/configuration-processors.html#condition-regexp

We actually use it in system.yml example, check:


(Robin Guo) #3

Hi @exekias,

with your instructions, we got more mount_point that we expected.

the metricbeat configure file

/etc/metricbeat/metricbeat.yml

#================================ Processors ===================================
processors:
  - drop_fields:
      fields: ['metricset.rtt','beat.version','beat.hostname','_score','system.cpu.irq.pct','system.cpu.steal.pct','system.load.norm.1',
              'system.load.norm.5','system.load.norm.15','system.cpu.softirq.pct','system.memory.free','system.memory.used.pct',
              'system.memory.used.bytes','system.process.ppid','system.process.username','system.process.pgid','system.process.cpu.start_time']
      
  - drop_event.when.regexp:
      system.filesytem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib)($|/)'

the results from metricbeat debug mode

 "dc": "fr4",
  "env": "production",
  "metricset": {
    "module": "system",
    "name": "filesystem"
  },
  "os": "linux",
  "service": "system",
  "system": {
    "filesystem": {
      "available": 0,
      "device_name": "cgroup",
      "files": 0,
      "free": 0,
      "free_files": 0,
      "mount_point": "/sys/fs/cgroup/memory",
      "total": 0,
      "used": {
        "bytes": 0,
        "pct": 0.000000
      }
    }
  },
  "tags": [
    "system"
  ],
  "type": "metricsets"
}
2017-07-26T17:57:50+08:00 DBG  Publish: {
  "@timestamp": "2017-07-26T09:57:50.289Z",
  "agent": "metricbeat",
  "beat": {
    "name": "fr4eslogstash02"
  },
  "dc": "fr4",
  "env": "production",
  "metricset": {
    "module": "system",
    "name": "filesystem"
  },
  "os": "linux",
  "service": "system",
  "system": {
    "filesystem": {
      "available": 0,
      "device_name": "cgroup",
      "files": 0,
      "free": 0,
      "free_files": 0,
      "mount_point": "/sys/fs/cgroup/cpuset",

the results from grafana dashboard.


(Robin Guo) #4

Any ideas on this issue?


(Carlos Pérez Aradros) #5

uhm, settings look good to me, did you restart metrcibeat after updating them?


(Robin Guo) #6

I did this, and also restart the logstash..
I'm using the metricbeat 5.4.3...

a complate the drop events as following:

- drop_event:
      when:
        or:
          - regexp:
              system.filesytem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib|run)'
          - regexp:
              system.filesytem.device_name: '^(selinuxfs|systemd-1|mqueue|debugfs|hugetlbfs|binfmt_misc|tmpfs|sysfs|proc|devtmpfs|securityfs|devpts|cgroup|pstore|configfs)'

(Robin Guo) #7

I used centos 7 with metricbeat version v5.4.3.

whatever any methods I tried, it doesn't works with metricbeat drop event rule.

:sob: :sob: :sob: :sob:


(Robin Guo) #8

@exekias
Can you give me more options or more drop events rule?


(Tudor Golubenco) #9

Just to troubleshoot, can you start with a regexp that will match almost anything, just to make sure the processors are applied? Something like:

processors:
- drop_event.when.regexp:
    system.filesystem.mount_point: '.*'

Does this drop the events for you?


(Robin Guo) #10

hi @tudor,

Finally, the problem is fixed with filters or processors to drop the events.

Thanks


(system) #11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.