Metricbeat Missing Data

Elastic Stack Beats
1

I have Metricbeat receiving data from Perfmon counters, including Process > % Processor Time. this is configured to collect every 10 seconds. However, we are missing processes for some reason. In other words, the events are not showing all the processes that we know are running on the machine at a given time. When we look in Perfmon directly, we can see the missing processes, but not in Kibana. Is there a reason Metricbeat is missing these data? At first we thought it was due to the 10s setting, but the process is running constantly and should be picked up at one of these intervals. It is crucial that we not miss any information - for troubleshooting reasons.

2

Hi @Tim_Mobley

Please share

modules.d/system.yml

and look at this in detail

There are many options... the default collects the Top 5, there are comfigurations to collect more, specific etc..etc..

process.include_top_n
These options allow you to filter out all processes that are not in the top N by CPU or memory, in order to reduce the number of documents created. If both the by_cpu and by_memory options are used, the union of the two sets is included.

and the default is the following...

  process.include_top_n:
    by_cpu: 5      # include top 5 processes by CPU
    by_memory: 5   # include top 5 processes by memory

So you will probably want to adjust / take out etc..

If you take it out you pretty sure you will get every process and it will be quite verbose.

Also you can use filtering

processes

When the process metricset is enabled, you can use the processes option to define a list of regexp expressions to filter the processes that are reported. For more complex filtering, you should use the processors configuration option. See Processors for more information.

1 Like
3

Here's my system.yml:

- module: system
  period: 10s
  metricsets:
    - cpu
    - core
    - memory
    - network
    - process
    - process_summary
    - socket_summary
  process.include_cpu_ticks: true
  process.include_top_n:
    by_cpu: 10
    by_memory: 10

- module: system
  period: 1m
  metricsets: 
    - filesystem
    - fsstat
  processors: 
    - drop_event.when.regexp:
        system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib|snap)(%|/)'

- module: system
  period: 15m
  metricsets:
    - uptime

So, if I wanted to see all the processes not matter what their utilization, should I just omit the process.include_top_n section?

4

Yup!

BTW I see no diskio.. :wink:

1 Like
5

Do you recommend that monitoring diskio metric as a standard practice?

6

Depends on what is important to you... Some of the default dashboards have diskio metrics so those viz will be empty if diskio is not on...

But if diskio is not important (say for stateless apps) then no need

2 Likes
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.