I have configured metricbeat to check the process status. I have noticed that it doesn't get the process info sometimes even if the process was running. It is possible that the process was not active some times but still running.
What could be the possible reason ? Please suggest.
So what may be happening is that you are using the default system.yml (system module) see with the defaults for the process metric set so you are probably using the Top N settings and sometimes the process you are looking for is in the Top 5 other times it is not
- module: system
period: 10s
metricsets:
- cpu
- load
- memory
- network
- process
- process_summary
- socket_summary
#- entropy
#- core
#- diskio
#- socket
#- service
#- users
process.include_top_n:
by_cpu: 5 # include top 5 processes by CPU
by_memory: 5 # include top 5 processes by memory
Above are the default settings
process.include_top_n
These options allow you to filter out all processes that are not in the top N by CPU or memory, in order to reduce the number of documents created. If both the by_cpu and by_memory options are used, the union of the two sets is included.
If you want all processes take out the top and use something like this
processes
When the process metricset is enabled, you can use the processes option to define a list of regexp expressions to filter the processes that are reported. For more complex filtering, you should use the processors configuration option. See Processors for more information.
The following example config returns metrics for all processes:
processes: ['.*']
Careful that might return A LOT of data ...
So you can adjust there are many settings to adjust it depends on what you want.
Thank you so much for the response.
I need to see top 10 processes consuming the most resources + data for some specific processes always. Can I achieve that.
I see am option to set the regex for specific processes. But then I will get information about only those processes, right ? Although there could be some other process which consume lot of resources .
Hi @xyz2
You can do both... there are probably a couple ways but here is a way you just set up to separate process collectors. First one does top N, second looks for specific processes.
Give it a try ... let us know
- module: system
period: 10s
metricsets: ['process']
process.include_top_n:
by_cpu: 5 # include top 5 processes by CPU
by_memory: 5 # include top 5 processes by memory
- module: system
period: 10s
metricsets: ['process']
processes: ['^sysmon*']
thanks for the response. I will try this.
Have a question. Adding this will also require a change in the current metrcibeat dashboard, right? In the dashboard, I only need top 10 processes as per first collector.
I don't think it will affect the dashboard if you leave the Top N (10) it should be fine... more data will not bother the dashboards AND I encourage you to play with the dashboards... just Edit -> Save As New and play with it!
I have multiple different processes by same process name . In my query I use process.args to look for my specific process. I specified the process name in the "processes" property as per your suggestion but still it doesn't pick that process every 30s (as defined). I disabled process.include_top_n and tried but no luck.
What am I missing ?
Also how can I differentiate whether process info is coming from process collector 1 or 2 ?
BTW I just tested this and it works as expected and each set is tagged appropriately please share your config and query. Also if the processes you are looking for are owned by root then you need to run metricbeat as root or with the correct privleges
- module: system
period: 10s
metricsets: ['process']
tags: ["myspecialprocess"]
processes: ['^md*']
- module: system
period: 10s
metricsets: ['process']
tags: ["mytopprocess"]
process.include_top_n:
by_cpu: 5 # include top 5 processes by CPU
by_memory: 5 # include top 5 processes by memory
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.