# JVM metrics failing on latest java17 openj9 with the java agent

**URL:** <https://discuss.elastic.co/t/jvm-metrics-failing-on-latest-java17-openj9-with-the-java-agent/337863>\
**Category:** APM\
**Tags:** docker, java\
**Created:** [July 7, 2023, 7:30am UTC](https://discuss.elastic.co/t/jvm-metrics-failing-on-latest-java17-openj9-with-the-java-agent/337863 "2023-07-07T07:30:34Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![kroonprins](https://avatars.discourse-cdn.com/v4/letter/k/7ea924/32.png) [@kroonprins](https://discuss.elastic.co/u/kroonprins)\
**Post date:** [July 7, 2023, 7:30am UTC](https://discuss.elastic.co/t/jvm-metrics-failing-on-latest-java17-openj9-with-the-java-agent/337863/1 "2023-07-07T07:30:35Z")

</div>

**APM Agent language and version** : java 1.39.0

**Description of the problem including expected versus actual behavior** :

When using the latest version of the openj9 docker image for java 17 (bm-semeru-runtimes:open-17-jdk-jammy@sha256:cd1e06810b6cb7c05efd6697135077cbd0801123b24100a9a0c24ffb332c6a8a), the elastic agent logs this error and jvm related metrics are not sent to the apm server:

```auto
2023-07-07 07:22:50,395 [elastic-apm-shared] ERROR co.elastic.apm.agent.util.ExecutorUtils - null
java.lang.UnsupportedOperationException: null
	at com.ibm.lang.management.internal.ExtendedThreadMXBeanImpl.getThreadAllocatedBytes(ExtendedThreadMXBeanImpl.java:82) ~[?:?]
	at co.elastic.apm.agent.metrics.builtin.JvmGcMetrics$HotspotAllocationSupplier.get(JvmGcMetrics.java:82) ~[elastic-apm-agent.jar:1.39.0]
	at co.elastic.apm.agent.report.serialize.MetricRegistrySerializer.serializeGauges(MetricRegistrySerializer.java:125) ~[elastic-apm-agent.jar:1.39.0]
	at co.elastic.apm.agent.report.serialize.MetricRegistrySerializer.serialize(MetricRegistrySerializer.java:92) ~[elastic-apm-agent.jar:1.39.0]
	at co.elastic.apm.agent.report.serialize.MetricRegistrySerializer.serialize(MetricRegistrySerializer.java:76) ~[elastic-apm-agent.jar:1.39.0]
	at co.elastic.apm.agent.report.serialize.MetricRegistrySerializer.serialize(MetricRegistrySerializer.java:56) ~[elastic-apm-agent.jar:1.39.0]
	at co.elastic.apm.agent.report.serialize.MetricRegistryReporter.report(MetricRegistryReporter.java:74) ~[elastic-apm-agent.jar:1.39.0]
	at co.elastic.apm.agent.metrics.MetricRegistry.flipPhaseAndReport(MetricRegistry.java:201) ~[elastic-apm-agent.jar:1.39.0]
	at co.elastic.apm.agent.report.serialize.MetricRegistryReporter.run(MetricRegistryReporter.java:59) ~[elastic-apm-agent.jar:1.39.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at co.elastic.apm.agent.util.ExecutorUtils$2.run(ExecutorUtils.java:99) [elastic-apm-agent.jar:1.39.0]
	at java.lang.Thread.run(Thread.java:857) [?:?]

```

Using a previous image, e.g. ibm-semeru-runtimes:open-17-jdk-jammy@sha256:28f0d18fb8a9444cc6f4d34bfd338fe871558bc4454a5ed2c90b44949d5aae03, then this error does not happen en jvm related metrics do get sent to the apm server.

To reproduce (in the example I use a dummy apm server and a dummy spring boot application):

```auto
$ docker run --rm -v $(pwd)/work:/work -e ELASTIC_APM_SERVICE_NAME=test -e ELASTIC_APM_SERVICE_VERSION=1 -e ELASTIC_APM_SERVER_URLS=http://example.com -e ELASTIC_APM_ENABLED=true -e ELASTIC_APM_LOG_LEVEL=INFO -e ELASTIC_APM_SECRET_TOKEN=test ibm-semeru-runtimes:open-17-jdk-jammy@sha256:cd1e06810b6cb7c05efd6697135077cbd0801123b24100a9a0c24ffb332c6a8a java -javaagent:/work/elastic-apm-agent.jar -jar /work/demo-0.0.1-SNAPSHOT.jar
2023-07-07 07:24:55,875 [main] INFO co.elastic.apm.agent.util.JmxUtils - Found JVM-specific OperatingSystemMXBean interface: com.ibm.lang.management.OperatingSystemMXBean
2023-07-07 07:24:55,900 [main] INFO co.elastic.apm.agent.util.JmxUtils - Found JVM-specific ThreadMXBean interface: com.sun.management.ThreadMXBean
2023-07-07 07:24:55,909 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM 1.39.0 as test (1) on Java 17.0.7 Runtime version: 17.0.7+7 VM version: openj9-0.38.0 (IBM Corporation) Linux 5.15.0-75-generic
2023-07-07 07:24:55,910 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - service_name: 'test' (source: Environment Variables)
2023-07-07 07:24:55,910 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - service_version: '1' (source: Environment Variables)
2023-07-07 07:24:55,910 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - secret_token: 'XXXX' (source: Environment Variables)
2023-07-07 07:24:55,910 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - server_urls: 'http://example.com' (source: Environment Variables)
2023-07-07 07:24:55,910 [main] WARN co.elastic.apm.agent.configuration.StartupInfo - To enable all features and decrease startup time, please configure application_packages
2023-07-07 07:24:56,186 [main] INFO co.elastic.apm.agent.bci.ElasticApmAgent - Overriding IndyBootstrapDispatcher class's module to java.base module. This is required in J9 VMs.
2023-07-07 07:24:56,189 [main] INFO co.elastic.apm.agent.bci.ElasticApmAgent - Overriding IndyBootstrapDispatcher class's module to java.base module. This is required in J9 VMs.
2023-07-07 07:24:56,978 [main] INFO co.elastic.apm.agent.impl.ElasticApmTracer - Tracer switched to RUNNING state
2023-07-07 07:24:57,354 [elastic-apm-server-healthcheck] INFO co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is available: <!doctype html><html><head> <title>Example Domain</title> <meta charset="utf-8" /> <meta http-equiv="Content-type" content="text/html; charset=utf-8" /> <meta name="viewport" content="width=device-width, initial-scale=1" /> <style type="text/css"> body { background-color: #f0f0f2; margin: 0; padding: 0; font-family: -apple-system, system-ui, BlinkMacSystemFont, "Segoe UI", "Open Sans", "Helvetica Neue", Helvetica, Arial, sans-serif; } div { width: 600px; margin: 5em auto; padding: 2em; background-color: #fdfdff; border-radius: 0.5em; box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02); } a:link, a:visited { color: #38488f; text-decoration: none; } @media (max-width: 700px) { div { margin: 0 auto; width: auto; } } </style> </head><body><div> <h1>Example Domain</h1> <p>This domain is for use in illustrative examples in documents. You may use this domain in literature without prior coordination or asking for permission.</p> <p><a href="https://www.iana.org/domains/example">More information...</a></p></div></body></html>
2023-07-07 07:24:57,356 [elastic-apm-server-healthcheck] WARN co.elastic.apm.agent.report.ApmServerHealthChecker - Failed to parse version of APM server http://example.com/: Expecting '{' as object start. Found < at position: 1, following: `<`

  . _______ _ _
 /\\ / ___'___ _ _(_)_ ____ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/ ___)| |_)| | | | | || (_| | ) ) ) )
  ' | ____| .__ |_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot :: (v3.1.1)

2023-07-07T07:24:57.939Z INFO 1 --- [main] com.example.demo.DemoApplication : Starting DemoApplication v0.0.1-SNAPSHOT using Java 17.0.7 with PID 1 (/work/demo-0.0.1-SNAPSHOT.jar started by root in /)
2023-07-07T07:24:57.946Z INFO 1 --- [main] com.example.demo.DemoApplication : No active profile set, falling back to 1 default profile: "default"
2023-07-07T07:24:59.128Z INFO 1 --- [main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)
2023-07-07T07:24:59.139Z INFO 1 --- [main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2023-07-07T07:24:59.139Z INFO 1 --- [main] o.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/10.1.10]
2023-07-07T07:24:59.240Z INFO 1 --- [main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2023-07-07T07:24:59.242Z INFO 1 --- [main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 1210 ms
2023-07-07T07:24:59.747Z INFO 1 --- [main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path ''
2023-07-07T07:24:59.756Z INFO 1 --- [main] com.example.demo.DemoApplication : Started DemoApplication in 2.393 seconds (process running for 4.307)
2023-07-07 07:25:26,979 [elastic-apm-shared] ERROR co.elastic.apm.agent.util.ExecutorUtils - null
java.lang.UnsupportedOperationException: null
	at com.ibm.lang.management.internal.ExtendedThreadMXBeanImpl.getThreadAllocatedBytes(ExtendedThreadMXBeanImpl.java:82) ~[?:?]
	at co.elastic.apm.agent.metrics.builtin.JvmGcMetrics$HotspotAllocationSupplier.get(JvmGcMetrics.java:82) ~[elastic-apm-agent.jar:1.39.0]
	at co.elastic.apm.agent.report.serialize.MetricRegistrySerializer.serializeGauges(MetricRegistrySerializer.java:125) ~[elastic-apm-agent.jar:1.39.0]
	at co.elastic.apm.agent.report.serialize.MetricRegistrySerializer.serialize(MetricRegistrySerializer.java:92) ~[elastic-apm-agent.jar:1.39.0]
	at co.elastic.apm.agent.report.serialize.MetricRegistrySerializer.serialize(MetricRegistrySerializer.java:76) ~[elastic-apm-agent.jar:1.39.0]
	at co.elastic.apm.agent.report.serialize.MetricRegistrySerializer.serialize(MetricRegistrySerializer.java:56) ~[elastic-apm-agent.jar:1.39.0]
	at co.elastic.apm.agent.report.serialize.MetricRegistryReporter.report(MetricRegistryReporter.java:74) ~[elastic-apm-agent.jar:1.39.0]
	at co.elastic.apm.agent.metrics.MetricRegistry.flipPhaseAndReport(MetricRegistry.java:201) ~[elastic-apm-agent.jar:1.39.0]
	at co.elastic.apm.agent.report.serialize.MetricRegistryReporter.run(MetricRegistryReporter.java:59) ~[elastic-apm-agent.jar:1.39.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at co.elastic.apm.agent.util.ExecutorUtils$2.run(ExecutorUtils.java:99) [elastic-apm-agent.jar:1.39.0]
	at java.lang.Thread.run(Thread.java:857) [?:?]

```

With the a previous image:

```auto
2023-07-07 07:26:56,907 [main] INFO co.elastic.apm.agent.util.JmxUtils - Found JVM-specific OperatingSystemMXBean interface: com.ibm.lang.management.OperatingSystemMXBean
2023-07-07 07:26:56,953 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM 1.39.0 as test (1) on Java 17.0.6 Runtime version: 17.0.6+10 VM version: openj9-0.36.0 (IBM Corporation) Linux 5.15.0-75-generic
2023-07-07 07:26:56,954 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - service_name: 'test' (source: Environment Variables)
2023-07-07 07:26:56,954 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - service_version: '1' (source: Environment Variables)
2023-07-07 07:26:56,954 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - secret_token: 'XXXX' (source: Environment Variables)
2023-07-07 07:26:56,954 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - server_urls: 'http://example.com' (source: Environment Variables)
2023-07-07 07:26:56,954 [main] WARN co.elastic.apm.agent.configuration.StartupInfo - To enable all features and decrease startup time, please configure application_packages
2023-07-07 07:26:57,213 [main] INFO co.elastic.apm.agent.bci.ElasticApmAgent - Overriding IndyBootstrapDispatcher class's module to java.base module. This is required in J9 VMs.
2023-07-07 07:26:57,216 [main] INFO co.elastic.apm.agent.bci.ElasticApmAgent - Overriding IndyBootstrapDispatcher class's module to java.base module. This is required in J9 VMs.
2023-07-07 07:26:58,072 [main] INFO co.elastic.apm.agent.impl.ElasticApmTracer - Tracer switched to RUNNING state
2023-07-07 07:26:58,444 [elastic-apm-server-healthcheck] INFO co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is available: <!doctype html><html><head> <title>Example Domain</title> <meta charset="utf-8" /> <meta http-equiv="Content-type" content="text/html; charset=utf-8" /> <meta name="viewport" content="width=device-width, initial-scale=1" /> <style type="text/css"> body { background-color: #f0f0f2; margin: 0; padding: 0; font-family: -apple-system, system-ui, BlinkMacSystemFont, "Segoe UI", "Open Sans", "Helvetica Neue", Helvetica, Arial, sans-serif; } div { width: 600px; margin: 5em auto; padding: 2em; background-color: #fdfdff; border-radius: 0.5em; box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02); } a:link, a:visited { color: #38488f; text-decoration: none; } @media (max-width: 700px) { div { margin: 0 auto; width: auto; } } </style> </head><body><div> <h1>Example Domain</h1> <p>This domain is for use in illustrative examples in documents. You may use this domain in literature without prior coordination or asking for permission.</p> <p><a href="https://www.iana.org/domains/example">More information...</a></p></div></body></html>
2023-07-07 07:26:58,447 [elastic-apm-server-healthcheck] WARN co.elastic.apm.agent.report.ApmServerHealthChecker - Failed to parse version of APM server http://example.com/: Expecting '{' as object start. Found < at position: 1, following: `<`

  . _______ _ _
 /\\ / ___'___ _ _(_)_ ____ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/ ___)| |_)| | | | | || (_| | ) ) ) )
  ' | ____| .__ |_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot :: (v3.1.1)

2023-07-07T07:26:59.057Z INFO 1 --- [main] com.example.demo.DemoApplication : Starting DemoApplication v0.0.1-SNAPSHOT using Java 17.0.6 with PID 1 (/work/demo-0.0.1-SNAPSHOT.jar started by root in /)
2023-07-07T07:26:59.067Z INFO 1 --- [main] com.example.demo.DemoApplication : No active profile set, falling back to 1 default profile: "default"
2023-07-07T07:27:00.136Z INFO 1 --- [main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)
2023-07-07T07:27:00.149Z INFO 1 --- [main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2023-07-07T07:27:00.149Z INFO 1 --- [main] o.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/10.1.10]
2023-07-07T07:27:00.221Z INFO 1 --- [main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2023-07-07T07:27:00.222Z INFO 1 --- [main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 1090 ms
2023-07-07T07:27:00.722Z INFO 1 --- [main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path ''
2023-07-07T07:27:00.735Z INFO 1 --- [main] com.example.demo.DemoApplication : Started DemoApplication in 2.272 seconds (process running for 4.254)
2023-07-07 07:27:38,749 [elastic-apm-server-reporter] WARN co.elastic.apm.agent.report.AbstractIntakeApiHandler - Response body: <?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
	<head>
		<title>404 - Not Found</title>
	</head>
	<body>
		<h1>404 - Not Found</h1>
		<script type="text/javascript" src="//obj.ac.bcon.ecdns.net/ec_tpm_bcon.js"></script>
	</body>
</html>

```

Any idea if this is an issue with the agent or if something needs to be raised with openj9?

---

<div class="post-metadata">

**Author:** ![Sylvain\_Juge](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylvain_juge/32/55521_2.png) [@Sylvain\_Juge](https://discuss.elastic.co/u/Sylvain_Juge)\
**Post date:** [July 7, 2023, 7:53am UTC](https://discuss.elastic.co/t/jvm-metrics-failing-on-latest-java17-openj9-with-the-java-agent/337863/2 "2023-07-07T07:53:05Z")

</div>

Hi !

Thanks for reporting this, this is definitely a bug in the agent, it seems the newer JVM does not support this way of collecting the JVM metrics (the agent has to rely on a few implementation details that might have changed between the two versions).

I have opened [this issue](https://github.com/elastic/apm-agent-java/issues/3224) to track and solve this.

In the mean time could you stick to the previous version as a work-around ?

---

<div class="post-metadata">

**Author:** ![kroonprins](https://avatars.discourse-cdn.com/v4/letter/k/7ea924/32.png) [@kroonprins](https://discuss.elastic.co/u/kroonprins)\
**Post date:** [July 7, 2023, 8:24am UTC](https://discuss.elastic.co/t/jvm-metrics-failing-on-latest-java17-openj9-with-the-java-agent/337863/3 "2023-07-07T08:24:51Z")

</div>

Thanks for the quick reply!  
Yes, we can keep using the previous version for now. We do try to keep this image as updated as possible for security fixes, so it wouldn't be ideal to do this for a significant time.

---

<div class="post-metadata">

**Author:** ![Sylvain\_Juge](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylvain_juge/32/55521_2.png) [@Sylvain\_Juge](https://discuss.elastic.co/u/Sylvain_Juge)\
**Post date:** [July 7, 2023, 3:36pm UTC](https://discuss.elastic.co/t/jvm-metrics-failing-on-latest-java17-openj9-with-the-java-agent/337863/4 "2023-07-07T15:36:15Z")

</div>

I've managed to reproduce and fix it with [this PR](https://github.com/elastic/apm-agent-java/pull/3225), if you want to try a snapshot version should be available in a few minutes once the build had completed in build artifacts here: [Ignore gc alloc metrics when unsupported · elastic/apm-agent-java@43998c5 · GitHub](https://github.com/elastic/apm-agent-java/actions/runs/5487794769?pr=3225)

---

<div class="post-metadata">

**Author:** ![kroonprins](https://avatars.discourse-cdn.com/v4/letter/k/7ea924/32.png) [@kroonprins](https://discuss.elastic.co/u/kroonprins)\
**Post date:** [July 7, 2023, 4:02pm UTC](https://discuss.elastic.co/t/jvm-metrics-failing-on-latest-java17-openj9-with-the-java-agent/337863/5 "2023-07-07T16:02:38Z")

</div>

great, I'll give it a try once the pipeline completes (seems to be running for over an hour already 🙂 )

---

<div class="post-metadata">

**Author:** ![kroonprins](https://avatars.discourse-cdn.com/v4/letter/k/7ea924/32.png) [@kroonprins](https://discuss.elastic.co/u/kroonprins)\
**Post date:** [July 7, 2023, 4:41pm UTC](https://discuss.elastic.co/t/jvm-metrics-failing-on-latest-java17-openj9-with-the-java-agent/337863/6 "2023-07-07T16:41:36Z")

</div>

works for me 👍. No more error and apm server receives jvm metrics.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 4, 2023, 4:42pm UTC](https://discuss.elastic.co/t/jvm-metrics-failing-on-latest-java17-openj9-with-the-java-agent/337863/7 "2023-08-04T16:42:22Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.