Bug: APM Java - Springboot, v1.39.0+ - No JVM Metrics Found

After reviewing the following post and appropriate GitHub issue, utilizing the Java APM agent on versions 1.39.0+ (including 1.40.0), JVM Metrics fail to display in the Metrics tab of APM. Below is an example of me swapping the agent versions to show the drop-off:

After reviewing APM log levels, no error logs are thrown, and debug logs are very verbose, making tacking this source issue down difficult. I can post debug logs on request.

Versions:
Java -
openjdk version "11.0.19" 2023-04-18
OpenJDK Runtime Environment (build 11.0.19+7-post-Ubuntu-0ubuntu122.04.1)
Springboot versions: 2.5.1, 3.x.x
OpenJDK 64-Bit Server VM (build 11.0.19+7-post-Ubuntu-0ubuntu122.04.1, mixed mode, sharing)

APM -
1.39.0, 1.40.0

Unfortunately we will need the debug-level agent logs here to help you further. You can use GitHub Gists to upload the logs.

I think the linked issue should be unrelated, as it is specific to IBM Semeru JDK17.

Great, here you go:

The logs indicate that metrics are actually being sent:

2023-07-25 08:45:06,685 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICSET_JSON_WRITER event (sequence 1)
2023-07-25 08:45:06,685 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICSET_JSON_WRITER event (sequence 2)
2023-07-25 08:45:06,685 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICSET_JSON_WRITER event (sequence 3)
2023-07-25 08:45:06,685 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICSET_JSON_WRITER event (sequence 4)
2023-07-25 08:45:06,685 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICSET_JSON_WRITER event (sequence 5)
2023-07-25 08:45:06,686 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICSET_JSON_WRITER event (sequence 6)
2023-07-25 08:45:06,686 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICSET_JSON_WRITER event (sequence 7)
2023-07-25 08:45:06,686 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICSET_JSON_WRITER event (sequence 8)
2023-07-25 08:45:06,686 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICSET_JSON_WRITER event (sequence 9)

To verify that those are actually the JVM metrics and that they make it to elasticsearch please do the following:

  • Start your application with apm agent 1.39.0 and leave it running for a while
  • In Kibana got to discover and
    • Select a time range of at least a few minutes within which your application was runnign with agent 1.39.0
    • Select the APM dataview
    • Enter the following filter: service.name:"javaPyro" and processor.event :"metric" and "system.cpu.total.norm.pct" : *

You should then get some documents containing the CPU usage if the metrics made it from the agent to elasticsearch.

Performed the above, there are no docs per the query. I have attached a query searching for any metrics, and then system.cpu docs to show the difference.

Here is the full query's result

Here is a snippet of the server side logs at that time:

This seems to be an APM-server issue related to some new JVM metrics which were added in the Java Agent 1.39.0. Which APM-server version are you using?

AM Server Version: 8.7.0

I've been able to reproduce the issue. The newly added JVM metrics are not correctly mapped by pre 8.9.0 APM server versions, causing the metricsets (which include other JVM metrics) to be dropped.

I would suggest you to either

  • Upgrade your APM-server to 8.9.0
  • Disable the "bad" metrics in the APM Agent via disable_metrics=jvm.fd.*,jvm.memory.non_heap.pool.*

Awesome, thanks for looking into it Jonas

Minor clarification: We didn't try reproducing the issue, but came up with an explanation.
Unfortunately that explanation doesn't seem to be valid:
I tried to actually reproduce the behaviour you are seeing with an 8.7.0 deployment and can't reproduce it.

It looks like your deployment of the apm-package is somehow incorrect / corrupted.
Nonetheless disable_metrics=jvm.fd.*,jvm.memory.non_heap.pool.* still should do the trick for you.

Hi, I set this
-Delastic.apm.disable_metrics=jvm.fd.*,jvm.memory.non_heap.pool.* but JVM metrics also not found in apm7.13.2 ,just only upgrade apm-server?

I tested a higher version of the APM server and the problem went away.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.