Drop_event on process (metric beat 5.6.2)

[Peter_Steenbergen]

Hi,

This is my config:

- module: system
  metricsets:
    - process
  enabled: true
  period: 60s
  processes: ['.*']
  process.cgroups.enabled: false
  processors:
    - drop_event.when.range:
      system.process.cpu.total.pct.lte: 1

I would like only the processes with a higher cpu than 1%. However it keeps pushing all the processes. What is the best way to do this in version 5.6.2?

[ruflin]

I wonder if the problem above is related to indentation. Could you try:

  processors:
    - drop_event.when.range:
        system.process.cpu.total.pct.lte: 1

And if that does not work, try:

  processors:
    - drop_event:
        when:
          range:
            system.process.cpu.total.pct.lte: 1

Both should be identical.

[Peter_Steenbergen]

To bad, it still gives me back 147 processes, all of them are around 0.01%.
Any idea's? Can it be because of processes: ['.*'] entry. I tried removing it, but then I get no output whatsoever.

I am using version 5.6.2 but both both does not seem to trigger.
My full list is this one.

#==========================  Modules configuration ============================
metricbeat.modules:

#------------------------------- System Module -------------------------------
- module: system
  metricsets:
    - cpu
    - core
    - memory
    - network
  enabled: true
  period: 300s

- module: system
  metricsets:
    - filesystem
    - fsstat
    - diskio
  enabled: true
  period: 10m

- module: system
  metricsets:
    - load
  enabled: true
  period: 60s

- module: system
  metricsets:
    - process
  enabled: true
  period: 60s
  processes: ['.*']
  process.cgroups.enabled: false
  processors:
    - drop_event:
      when:
        range:
          system.process.cpu.total.pct.lte: 1

#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

#----------------------------- Logstash output --------------------------------
output.logstash:

[Peter_Steenbergen]

This is the output when I run with the parameter -e.
INFO Non-zero metrics in the last 30s: fetches.system-core.events=2 fetches.system-core.success=1 fetches.system-cpu.events=1 fetches.system-cpu.success=1 fetches.system-diskio.events=2 fetches.system-diskio.success=1 fetches.system-filesystem.events=31 fetches.system-filesystem.success=1 fetches.system-fsstat.events=1 fetches.system-fsstat.success=1 fetches.system-load.events=1 fetches.system-load.success=1 fetches.system-memory.events=1 fetches.system-memory.success=1 fetches.system-network.events=2 fetches.system-network.success=1 fetches.system-process.events=155 fetches.system-process.success=1 libbeat.logstash.call_count.PublishEvents=1 libbeat.logstash.publish.read_bytes=5137 libbeat.logstash.publish.write_bytes=15700 libbeat.logstash.published_and_acked_events=196 libbeat.publisher.messages_in_worker_queues=196 libbeat.publisher.published_events=196

I can see that it fetches 155 processes but no processors (filter) is trigger so it seems.
I put a '#' before processes: ['.*'], and keeps pushing all the processes. With monitoring several hundreds of servers it will take our disk space very fast.

[steffens (Steffen Siering)]

The processors configuration the way you use it has been added to 6.0 release only. In 5.6 it works a little different. You have to use:

- module: system
  ...
  filters:
    - drop_event.when.range:
        cpu.total.pct.lte: 1

The problem with 5.6 is, filtering happens on the not yet fully constructed event. That is, I'm not sure if you have to use system.process.cpu.total...., or just process.cpu.total... or just cpu.total..., or total...

[Peter_Steenbergen]

I will try the filter. I got my code from the documentation from 5.6 as I can remember.
Will come back when I try all the options.

[steffens (Steffen Siering)]

I will try the filter. I got my code from the documentation from 5.6 as I can remember.

I see. This can very well be the case. But the processors setting in 5.6 is applied globally only, not per module. You have to remove all indentation to have something like:

metricbeat.modules:
- module: ...
  ...
- module: ...

processors:
- drop_event.when.range: ...

[Peter_Steenbergen]

Ah oké, I did not noticed that processors are global. Have added the filter, and can see ik the morning if it works.

Then only another one for memory more then 30%.

Without the filters it kills the diskspace with several hundred servers.

[ruflin]

@Peter_Steenbergen I think @steffens is right. Sorry that I missed that in my initial answer.

[Peter_Steenbergen]

No problem. In the end this was the solution for me:

- module: system
  metricsets:
    - process
  enabled: true
  period: 15s
  process.cgroups.enabled: false
  filters:
    - drop_event.when.range:
        cpu.total.pct.lte: 0.10
        
- module: system
  metricsets:
    - process
  enabled: true
  period: 15s
  process.cgroups.enabled: false
  filters:
    - drop_event.when.range:
        memory.rss.pct.lte: 0.10

I found out though that 0.10 stands for 10%, 1 is 100%. So 1% would be 0.01. So now all my system events per 15s are being being processed when memory is 10% or cpu is 10% or more. Maybe I can chain it, but for now this would suffice.

Or do you guys have a solution for that?