How to fetch %cpu userspace using metricbeat modules?

[vschakka (Sukumar Chakka)]

Hi,
I have metric beat configuration as below:
metricbeat.modules:

• module: system
  metricsets: [cpu]
  cpu.metrics: [percentages, normalized_percentages]
  enabled: true
  period: 20s
  processes: ['.*']
  fields_under_root: true
  fields:
  df_env: test
  df_portal: xxx
  df_application: metric

output.console:
pretty: true

When i see the code of beats/metricbeat/module/system/cpu/helper.go
//helper.go
calculate := func(field2 uint64, field1 uint64) float64 {

		perc := 0.0
		delta := int64(field2 - field1)
		perc = float64(delta) / float64(allDelta)
		return system.Round(perc*float64(NumCPU), .5, 4)
	}

	current.UserPercent = calculate(current.Cpu.User, last.Cpu.User)
	current.SystemPercent = calculate(current.Cpu.Sys, last.Cpu.Sys)

Based on the above code I realized that while calculating percentage of fields it is using previous/last statistics like system.cpu.system.pct or system.cpu.user.pct fields. But what I actually need is the value**(%Cpu(s): 2.6 us)** which is displaying at top command as below

//command output when i run the top command.
$> top
Tasks: 239 total, 1 running, 238 sleeping, 0 stopped, 0 zombie
%Cpu(s): 2.6 us, 0.8 sy, 0.0 ni, 96.6 id, 0.0 wa, 0.0 hi, 0.1 si, 0.0 st
KiB Mem: 24522424 total, 11352344 used, 13170080 free, 517500 buffers
KiB Swap: 3145724 total, 0 used, 3145724 free. 1916320 cached Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6956 root 20 0 9231628 1.234g 34144 S 19.6 5.3 2677:23 java
31918 gdev 20 0 7798272 1.194g 16060 S 6.3 5.1 900:06.61 java
31959 gdev 20 0 7798272 1.181g 16044 S 5.3 5.0 914:09.06 java
32436 gdev 20 0 7798272 1.148g 16056 S 5.3 4.9 932:21.06 java
31877 gdev 20 0 7798272 1.183g 16032 S 5.0 5.1 902:22.92 java
3629 root 20 0 598044 30272 9348 S 2.3 0.1 272:01.77 BESClient
20392 gdev 20 0 123664 1744 1148 R 1.0 0.0 0:00.06 top

I need to create alert configuration when cpu system.cpu.system.pct > 80%.
But this value is always less than 1% as it is using last metrics.

Please help me how to retrieve above value . Thanks in advance

[jsoriano (Jaime Soriano)]

Hi @vschakka,

CPU metrics are sent as ratios, this could be the reason why you see them with values under 1.
The equivalent to the % of CPU in user space that you see in the top command is stored in the system.cpu.user.norm.pct, I see you already are collecting normalized_percentages, so you should have this metric available.
So if you want to create an alert that triggers when the userspace CPU of the host is over 80% you have to create an alert as system.cpu.user.norm.pct > 0.8.

[vschakka (Sukumar Chakka)]

Thanks for your reply.
There are many java processes running in my VM. I need a particular java process cpu utilization . So that i can identify which java process is taking more cpu power and set alerts accordingly. My specific java processes have certain special strings when i do ps -ef in the command line (For instance prod_abc_def_4) which can be used to filter amongst the list.

And I highlighted below %cpu which i need in the top command output:
$> top
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6956 root 20 0 9231628 1.234g 34144 S 19.6 5.3 2677:23 java
31918 gdev 20 0 7798272 1.194g 16060 S 6.3 5.1 900:06.61 java
31959 gdev 20 0 7798272 1.181g 16044 S 5.3 5.0 914:09.06 java
32436 gdev 20 0 7798272 1.148g 16056 S 5.3 4.9 932:21.06 java
31877 gdev 20 0 7798272 1.183g 16032 S 5.0 5.1 902:22.92 java

metricbeat configuration: ( I am not sure it is right)
metricbeat.modules:
- module: system
metricsets: ["process"]
enabled: true
period: 20s
processes: ['prod_abc_def_4']
process.cgroups.enabled: false
process.cmdline.cache.enabled: true
fields_under_root: true
fields:
df_env: test
df_portal: dcim
df_application: metric

output.console:
  pretty: true

Please help with correct metric configuration to fetch %cpu usage(shown and highlighted in top command above) of a particular java process. Thanks in advance.

[jsoriano (Jaime Soriano)]

processes setting filter processes by their names, in the case of java processes their name is just java, so prod_abc_def_4 don't match any process.

If you only want to send metrics of processes that contain prod_abd_def_4 in their command line, you can try dropping the rest of events with the drop_event processor, something like:

processors:
 - drop_event:
     when:
        not:
           contains:
               system.process.cmdline: "prod_abc_def_4"

Another option could be to send the metrics of all events, or of all events of java processes, and filter later when querying.

[vschakka (Sukumar Chakka)]

Thanks. It worked perfectly.