Kibana version: 7.17.10
Elasticsearch version: 7.17.10
APM Server version: 7.17.10
APM Agent language and version: Java agent, 1.45.0
Java version: Java 21
Original install method (e.g. download page, yum, deb, from source, etc.) and version: Elastic Stack installed through Elasticsearch (ECK) Operator 2.7.0, Agents embedded in Docker images running in K8s
Is there anything special in your setup? No
The issue is that we have migrated some services to Java 21 from Java 17, and all those services are reporting now in the APM a System CPU base level of 20-30%, even those that are doing basically nothing. The "same" services in Java 17 had an iddle base level of 0.5-1.0% System CPU.
The only place where I can see such levels is in Elastic APM, as OpenShift monitoring, top or even Java Flight Recorder sessions do not show any differences in CPU usage.
I downgraded one of the services to Java 17 and ran it in a different namespace and this is the APM JVM CPU chart:
Number of threads, memory used (Heap and native), even process CPU is the same...
Some libraries have been upgraded, else the services don't work with Java 21, but VisualVM and JFR show no Java code consuming CPU so. The docker image is the same except downloading a different JDK.
I understand this is probably not an agent or APM problem but something that might have changed in Java when reporting those values, but I have been unable to find any reference about it, so what I'm asking is:
Has anybody else experienced this?
Is this a well known issue with Java 21?
Shall we just ignore system CPU in the JVMs?
Any idea what might be happening?
I'm trying to get the admins to add Metricbeat as DaemonSet to gather more and better data, but unfortunately we are not there yet