Java APM Agent, System CPU reporting with Java 21

Elastic Observability APM
,
1

Kibana version: 7.17.10

Elasticsearch version: 7.17.10

APM Server version: 7.17.10

APM Agent language and version: Java agent, 1.45.0

Java version: Java 21

Original install method (e.g. download page, yum, deb, from source, etc.) and version: Elastic Stack installed through Elasticsearch (ECK) Operator 2.7.0, Agents embedded in Docker images running in K8s

Is there anything special in your setup? No

The issue is that we have migrated some services to Java 21 from Java 17, and all those services are reporting now in the APM a System CPU base level of 20-30%, even those that are doing basically nothing. The "same" services in Java 17 had an iddle base level of 0.5-1.0% System CPU.

The only place where I can see such levels is in Elastic APM, as OpenShift monitoring, top or even Java Flight Recorder sessions do not show any differences in CPU usage.

I downgraded one of the services to Java 17 and ran it in a different namespace and this is the APM JVM CPU chart:
Java 17:

Java17
Java17812×327 35.8 KB

Java 21:
Java21
Java21815×327 22.3 KB

Number of threads, memory used (Heap and native), even process CPU is the same...
Some libraries have been upgraded, else the services don't work with Java 21, but VisualVM and JFR show no Java code consuming CPU so. The docker image is the same except downloading a different JDK.

I understand this is probably not an agent or APM problem but something that might have changed in Java when reporting those values, but I have been unable to find any reference about it, so what I'm asking is:

Has anybody else experienced this?
Is this a well known issue with Java 21?
Shall we just ignore system CPU in the JVMs?
Any idea what might be happening?

I'm trying to get the admins to add Metricbeat as DaemonSet to gather more and better data, but unfortunately we are not there yet :smiley:

Cheers,
D.

2

The reported metrics are simply the values returned from OperatingSystemMXBean.getSystemCPULoad() and OperatingSystemMXBean.getProcessCPULoad().

As a result, the actual metric values are highly dependendent on the JVM version. My guess in your case would be that with Java 17 the JVM somehow tried to guess the container CPU usage and reported that as system CPU, whereas for Java 21 it might be the actual CPU usage of the underlying host.

In general, system cpu usage shouldn't realy be relied on, you should use proper host monitoring instead. The process CPU usage however should be reliable.

1 Like
3

Hi Jonas,
Thank you. That's what I suspected but wanted to confirm that I was not going nuts.
Once Java 21 becomes the norm, still a way to go, I guess I won't be the only one wondering :smiley:

Thanks again!
D.

4

Yeah it's confusing for sure. We'd need to dig through the JVM changes to find the actual root cause of this behaviour.

5

I found some bugs related to the topic but they were solved in Java 14.

I was not able to check the differences, because the OperatingSystemMXBean class and the implementation classes like UnixOperatingSystemMXBean are com.sun.* and I could not find the source.

No mention of those in recent release notes either.
:man_shrugging:

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.