Can't drop events from metricbeat

Elastic Stack Beats
1

Hi folks,

We encountered an issue while the metricbeat is trying to send their metrics to elasticsearch.
even with some drop events rules, but can't drop the events we don't want. we still can see some particular values of a field.

metricbeat.yml

 - drop_event:
      when:
         or:
           - equals: 
                system.filesystem.device_name: "tmpfs"
           - equals: 
                system.filesystem.device_name: "devpts"
           - equals: 
                system.filesystem.device_name: "none"
           - equals: 
                system.filesystem.device_name: "sysfs"
           - equals: 
                system.filesystem.device_name: "proc"
           - equals: 
                system.filesystem.device_name: "cgroup"
           - equals: 
                system.filesystem.device_name: "hugetlbfs"
           - equals: 
                system.filesystem.device_name: "systemd-1"
           - equals: 
                system.filesystem.device_name: "devtmpfs"
           - equals: 
                system.filesystem.device_name: "debugfs"
           - equals: 
                system.filesystem.mount_point: "dev/shm"
           - equals: 
                system.filesystem.mount_point: "/dev/pts"
           - equals: 
                system.filesystem.mount_point: "/sys/fs/selinux"
           - equals: 
                system.filesystem.mount_point: "/proc"
           - equals: 
                system.filesystem.mount_point: "/proc/*"
           - equals:
                sytem.filesystem.mount_point: "/dev"
           - equals:  
                sytem.filesystem.mount_point: "/dev/hugepages"
           - equals:  
                sytem.filesystem.mount_point: "/dev/mqueue"
           - equals:  
                sytem.filesystem.mount_point: "/proc/sys/fs/binfmt_misc"
           - equals: 
                sytem.filesystem.mount_point: "/sys/fs/cgroup/blkio"
           - equals:  
                sytem.filesystem.mount_point: "/sys/fs/cgroup/cpu,cpuacct"
           - equals:  
                sytem.filesystem.mount_point: "/sys/fs/cgroup/cpuset"
           - equals:  
               sytem.filesystem.mount_point: "/sys/fs/cgroup/devices"

The results from grafana,but also can get the same results from kibana

image.png1890×452 52.5 KB

it would be great if there is a way can drop some events start with /sys/*/.... with a wildcard.

Thanks

2

It's possible, I think you want to check https://www.elastic.co/guide/en/beats/metricbeat/current/configuration-processors.html#condition-regexp

We actually use it in system.yml example, check:

3

Hi @exekias,

with your instructions, we got more mount_point that we expected.

the metricbeat configure file

/etc/metricbeat/metricbeat.yml

#================================ Processors ===================================
processors:
  - drop_fields:
      fields: ['metricset.rtt','beat.version','beat.hostname','_score','system.cpu.irq.pct','system.cpu.steal.pct','system.load.norm.1',
              'system.load.norm.5','system.load.norm.15','system.cpu.softirq.pct','system.memory.free','system.memory.used.pct',
              'system.memory.used.bytes','system.process.ppid','system.process.username','system.process.pgid','system.process.cpu.start_time']
      
  - drop_event.when.regexp:
      system.filesytem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib)($|/)'

the results from metricbeat debug mode

 "dc": "fr4",
  "env": "production",
  "metricset": {
    "module": "system",
    "name": "filesystem"
  },
  "os": "linux",
  "service": "system",
  "system": {
    "filesystem": {
      "available": 0,
      "device_name": "cgroup",
      "files": 0,
      "free": 0,
      "free_files": 0,
      "mount_point": "/sys/fs/cgroup/memory",
      "total": 0,
      "used": {
        "bytes": 0,
        "pct": 0.000000
      }
    }
  },
  "tags": [
    "system"
  ],
  "type": "metricsets"
}
2017-07-26T17:57:50+08:00 DBG  Publish: {
  "@timestamp": "2017-07-26T09:57:50.289Z",
  "agent": "metricbeat",
  "beat": {
    "name": "fr4eslogstash02"
  },
  "dc": "fr4",
  "env": "production",
  "metricset": {
    "module": "system",
    "name": "filesystem"
  },
  "os": "linux",
  "service": "system",
  "system": {
    "filesystem": {
      "available": 0,
      "device_name": "cgroup",
      "files": 0,
      "free": 0,
      "free_files": 0,
      "mount_point": "/sys/fs/cgroup/cpuset",

the results from grafana dashboard.

1898×388 28.3 KB

4

Any ideas on this issue?

5

uhm, settings look good to me, did you restart metrcibeat after updating them?

6

I did this, and also restart the logstash..
I'm using the metricbeat 5.4.3...

a complate the drop events as following:

- drop_event:
      when:
        or:
          - regexp:
              system.filesytem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib|run)'
          - regexp:
              system.filesytem.device_name: '^(selinuxfs|systemd-1|mqueue|debugfs|hugetlbfs|binfmt_misc|tmpfs|sysfs|proc|devtmpfs|securityfs|devpts|cgroup|pstore|configfs)'
7

I used centos 7 with metricbeat version v5.4.3.

whatever any methods I tried, it doesn't works with metricbeat drop event rule.

:sob: :sob: :sob: :sob:

8

@exekias
Can you give me more options or more drop events rule?

9

Just to troubleshoot, can you start with a regexp that will match almost anything, just to make sure the processors are applied? Something like:

processors:
- drop_event.when.regexp:
    system.filesystem.mount_point: '.*'

Does this drop the events for you?

10

hi @tudor,

Finally, the problem is fixed with filters or processors to drop the events.

Thanks

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.