Metricbeat doesn't recognize the process

[xyz2]

HI,

I have configured metricbeat to check the process status. I have noticed that it doesn't get the process info sometimes even if the process was running. It is possible that the process was not active some times but still running.

What could be the possible reason ? Please suggest.

[stephenb (Stephen Brown)]

Hi @xyz2

So what may be happening is that you are using the default system.yml (system module) see with the defaults for the process metric set so you are probably using the Top N settings and sometimes the process you are looking for is in the Top 5 other times it is not

- module: system
  period: 10s
  metricsets:
    - cpu
    - load
    - memory
    - network
    - process
    - process_summary
    - socket_summary
    #- entropy
    #- core
    #- diskio
    #- socket
    #- service
    #- users
  process.include_top_n:
    by_cpu: 5      # include top 5 processes by CPU
    by_memory: 5   # include top 5 processes by memory

Above are the default settings

process.include_top_n

These options allow you to filter out all processes that are not in the top N by CPU or memory, in order to reduce the number of documents created. If both the by_cpu and by_memory options are used, the union of the two sets is included.

If you want all processes take out the top and use something like this

processes

When the process metricset is enabled, you can use the processes option to define a list of regexp expressions to filter the processes that are reported. For more complex filtering, you should use the processors configuration option. See Processors for more information.

The following example config returns metrics for all processes:

processes: ['.*']

Careful that might return A LOT of data ...

So you can adjust there are many settings to adjust it depends on what you want.

[xyz2]

Thank you so much for the response.
I need to see top 10 processes consuming the most resources + data for some specific processes always. Can I achieve that.
I see am option to set the regex for specific processes. But then I will get information about only those processes, right ? Although there could be some other process which consume lot of resources .

[stephenb (Stephen Brown)]

Hi @xyz2
You can do both... there are probably a couple ways but here is a way you just set up to separate process collectors. First one does top N, second looks for specific processes.
Give it a try ... let us know

- module: system
  period: 10s
  metricsets: ['process']
  process.include_top_n:
    by_cpu: 5      # include top 5 processes by CPU
    by_memory: 5   # include top 5 processes by memory

- module: system
  period: 10s
  metricsets: ['process']
  processes: ['^sysmon*']

[xyz2]

thanks for the response. I will try this.
Have a question. Adding this will also require a change in the current metrcibeat dashboard, right? In the dashboard, I only need top 10 processes as per first collector.

[stephenb (Stephen Brown)]

I don't think it will affect the dashboard if you leave the Top N (10) it should be fine... more data will not bother the dashboards AND I encourage you to play with the dashboards... just Edit -> Save As New and play with it!

[xyz2]

Hi ,

I have multiple different processes by same process name . In my query I use process.args to look for my specific process. I specified the process name in the "processes" property as per your suggestion but still it doesn't pick that process every 30s (as defined). I disabled process.include_top_n and tried but no luck.

What am I missing ?

Also how can I differentiate whether process info is coming from process collector 1 or 2 ?

Appreciate your help !

[stephenb (Stephen Brown)]

Share your config please... and query please.

Let take out include_top_n for now and focus on the process with your specific name.

[stephenb (Stephen Brown)]

BTW I just tested this and it works as expected and each set is tagged appropriately please share your config and query. Also if the processes you are looking for are owned by root then you need to run metricbeat as root or with the correct privleges

- module: system
  period: 10s
  metricsets: ['process']
  tags: ["myspecialprocess"]
  processes: ['^md*']

- module: system
  period: 10s
  metricsets: ['process']
  tags: ["mytopprocess"]
  process.include_top_n:
    by_cpu: 5      # include top 5 processes by CPU
    by_memory: 5   # include top 5 processes by memory