APM Agent language and version: java 1.39.0
Description of the problem including expected versus actual behavior:
When using the latest version of the openj9 docker image for java 17 (bm-semeru-runtimes:open-17-jdk-jammy@sha256:cd1e06810b6cb7c05efd6697135077cbd0801123b24100a9a0c24ffb332c6a8a), the elastic agent logs this error and jvm related metrics are not sent to the apm server:
2023-07-07 07:22:50,395 [elastic-apm-shared] ERROR co.elastic.apm.agent.util.ExecutorUtils - null
java.lang.UnsupportedOperationException: null
at com.ibm.lang.management.internal.ExtendedThreadMXBeanImpl.getThreadAllocatedBytes(ExtendedThreadMXBeanImpl.java:82) ~[?:?]
at co.elastic.apm.agent.metrics.builtin.JvmGcMetrics$HotspotAllocationSupplier.get(JvmGcMetrics.java:82) ~[elastic-apm-agent.jar:1.39.0]
at co.elastic.apm.agent.report.serialize.MetricRegistrySerializer.serializeGauges(MetricRegistrySerializer.java:125) ~[elastic-apm-agent.jar:1.39.0]
at co.elastic.apm.agent.report.serialize.MetricRegistrySerializer.serialize(MetricRegistrySerializer.java:92) ~[elastic-apm-agent.jar:1.39.0]
at co.elastic.apm.agent.report.serialize.MetricRegistrySerializer.serialize(MetricRegistrySerializer.java:76) ~[elastic-apm-agent.jar:1.39.0]
at co.elastic.apm.agent.report.serialize.MetricRegistrySerializer.serialize(MetricRegistrySerializer.java:56) ~[elastic-apm-agent.jar:1.39.0]
at co.elastic.apm.agent.report.serialize.MetricRegistryReporter.report(MetricRegistryReporter.java:74) ~[elastic-apm-agent.jar:1.39.0]
at co.elastic.apm.agent.metrics.MetricRegistry.flipPhaseAndReport(MetricRegistry.java:201) ~[elastic-apm-agent.jar:1.39.0]
at co.elastic.apm.agent.report.serialize.MetricRegistryReporter.run(MetricRegistryReporter.java:59) ~[elastic-apm-agent.jar:1.39.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) ~[?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at co.elastic.apm.agent.util.ExecutorUtils$2.run(ExecutorUtils.java:99) [elastic-apm-agent.jar:1.39.0]
at java.lang.Thread.run(Thread.java:857) [?:?]
Using a previous image, e.g. ibm-semeru-runtimes:open-17-jdk-jammy@sha256:28f0d18fb8a9444cc6f4d34bfd338fe871558bc4454a5ed2c90b44949d5aae03, then this error does not happen en jvm related metrics do get sent to the apm server.
To reproduce (in the example I use a dummy apm server and a dummy spring boot application):
$ docker run --rm -v $(pwd)/work:/work -e ELASTIC_APM_SERVICE_NAME=test -e ELASTIC_APM_SERVICE_VERSION=1 -e ELASTIC_APM_SERVER_URLS=http://example.com -e ELASTIC_APM_ENABLED=true -e ELASTIC_APM_LOG_LEVEL=INFO -e ELASTIC_APM_SECRET_TOKEN=test ibm-semeru-runtimes:open-17-jdk-jammy@sha256:cd1e06810b6cb7c05efd6697135077cbd0801123b24100a9a0c24ffb332c6a8a java -javaagent:/work/elastic-apm-agent.jar -jar /work/demo-0.0.1-SNAPSHOT.jar
2023-07-07 07:24:55,875 [main] INFO co.elastic.apm.agent.util.JmxUtils - Found JVM-specific OperatingSystemMXBean interface: com.ibm.lang.management.OperatingSystemMXBean
2023-07-07 07:24:55,900 [main] INFO co.elastic.apm.agent.util.JmxUtils - Found JVM-specific ThreadMXBean interface: com.sun.management.ThreadMXBean
2023-07-07 07:24:55,909 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM 1.39.0 as test (1) on Java 17.0.7 Runtime version: 17.0.7+7 VM version: openj9-0.38.0 (IBM Corporation) Linux 5.15.0-75-generic
2023-07-07 07:24:55,910 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - service_name: 'test' (source: Environment Variables)
2023-07-07 07:24:55,910 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - service_version: '1' (source: Environment Variables)
2023-07-07 07:24:55,910 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - secret_token: 'XXXX' (source: Environment Variables)
2023-07-07 07:24:55,910 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - server_urls: 'http://example.com' (source: Environment Variables)
2023-07-07 07:24:55,910 [main] WARN co.elastic.apm.agent.configuration.StartupInfo - To enable all features and decrease startup time, please configure application_packages
2023-07-07 07:24:56,186 [main] INFO co.elastic.apm.agent.bci.ElasticApmAgent - Overriding IndyBootstrapDispatcher class's module to java.base module. This is required in J9 VMs.
2023-07-07 07:24:56,189 [main] INFO co.elastic.apm.agent.bci.ElasticApmAgent - Overriding IndyBootstrapDispatcher class's module to java.base module. This is required in J9 VMs.
2023-07-07 07:24:56,978 [main] INFO co.elastic.apm.agent.impl.ElasticApmTracer - Tracer switched to RUNNING state
2023-07-07 07:24:57,354 [elastic-apm-server-healthcheck] INFO co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is available: <!doctype html><html><head> <title>Example Domain</title> <meta charset="utf-8" /> <meta http-equiv="Content-type" content="text/html; charset=utf-8" /> <meta name="viewport" content="width=device-width, initial-scale=1" /> <style type="text/css"> body { background-color: #f0f0f2; margin: 0; padding: 0; font-family: -apple-system, system-ui, BlinkMacSystemFont, "Segoe UI", "Open Sans", "Helvetica Neue", Helvetica, Arial, sans-serif; } div { width: 600px; margin: 5em auto; padding: 2em; background-color: #fdfdff; border-radius: 0.5em; box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02); } a:link, a:visited { color: #38488f; text-decoration: none; } @media (max-width: 700px) { div { margin: 0 auto; width: auto; } } </style> </head><body><div> <h1>Example Domain</h1> <p>This domain is for use in illustrative examples in documents. You may use this domain in literature without prior coordination or asking for permission.</p> <p><a href="https://www.iana.org/domains/example">More information...</a></p></div></body></html>
2023-07-07 07:24:57,356 [elastic-apm-server-healthcheck] WARN co.elastic.apm.agent.report.ApmServerHealthChecker - Failed to parse version of APM server http://example.com/: Expecting '{' as object start. Found < at position: 1, following: `<`
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v3.1.1)
2023-07-07T07:24:57.939Z INFO 1 --- [ main] com.example.demo.DemoApplication : Starting DemoApplication v0.0.1-SNAPSHOT using Java 17.0.7 with PID 1 (/work/demo-0.0.1-SNAPSHOT.jar started by root in /)
2023-07-07T07:24:57.946Z INFO 1 --- [ main] com.example.demo.DemoApplication : No active profile set, falling back to 1 default profile: "default"
2023-07-07T07:24:59.128Z INFO 1 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)
2023-07-07T07:24:59.139Z INFO 1 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2023-07-07T07:24:59.139Z INFO 1 --- [ main] o.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/10.1.10]
2023-07-07T07:24:59.240Z INFO 1 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2023-07-07T07:24:59.242Z INFO 1 --- [ main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 1210 ms
2023-07-07T07:24:59.747Z INFO 1 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path ''
2023-07-07T07:24:59.756Z INFO 1 --- [ main] com.example.demo.DemoApplication : Started DemoApplication in 2.393 seconds (process running for 4.307)
2023-07-07 07:25:26,979 [elastic-apm-shared] ERROR co.elastic.apm.agent.util.ExecutorUtils - null
java.lang.UnsupportedOperationException: null
at com.ibm.lang.management.internal.ExtendedThreadMXBeanImpl.getThreadAllocatedBytes(ExtendedThreadMXBeanImpl.java:82) ~[?:?]
at co.elastic.apm.agent.metrics.builtin.JvmGcMetrics$HotspotAllocationSupplier.get(JvmGcMetrics.java:82) ~[elastic-apm-agent.jar:1.39.0]
at co.elastic.apm.agent.report.serialize.MetricRegistrySerializer.serializeGauges(MetricRegistrySerializer.java:125) ~[elastic-apm-agent.jar:1.39.0]
at co.elastic.apm.agent.report.serialize.MetricRegistrySerializer.serialize(MetricRegistrySerializer.java:92) ~[elastic-apm-agent.jar:1.39.0]
at co.elastic.apm.agent.report.serialize.MetricRegistrySerializer.serialize(MetricRegistrySerializer.java:76) ~[elastic-apm-agent.jar:1.39.0]
at co.elastic.apm.agent.report.serialize.MetricRegistrySerializer.serialize(MetricRegistrySerializer.java:56) ~[elastic-apm-agent.jar:1.39.0]
at co.elastic.apm.agent.report.serialize.MetricRegistryReporter.report(MetricRegistryReporter.java:74) ~[elastic-apm-agent.jar:1.39.0]
at co.elastic.apm.agent.metrics.MetricRegistry.flipPhaseAndReport(MetricRegistry.java:201) ~[elastic-apm-agent.jar:1.39.0]
at co.elastic.apm.agent.report.serialize.MetricRegistryReporter.run(MetricRegistryReporter.java:59) ~[elastic-apm-agent.jar:1.39.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) ~[?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at co.elastic.apm.agent.util.ExecutorUtils$2.run(ExecutorUtils.java:99) [elastic-apm-agent.jar:1.39.0]
at java.lang.Thread.run(Thread.java:857) [?:?]
With the a previous image:
2023-07-07 07:26:56,907 [main] INFO co.elastic.apm.agent.util.JmxUtils - Found JVM-specific OperatingSystemMXBean interface: com.ibm.lang.management.OperatingSystemMXBean
2023-07-07 07:26:56,953 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM 1.39.0 as test (1) on Java 17.0.6 Runtime version: 17.0.6+10 VM version: openj9-0.36.0 (IBM Corporation) Linux 5.15.0-75-generic
2023-07-07 07:26:56,954 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - service_name: 'test' (source: Environment Variables)
2023-07-07 07:26:56,954 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - service_version: '1' (source: Environment Variables)
2023-07-07 07:26:56,954 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - secret_token: 'XXXX' (source: Environment Variables)
2023-07-07 07:26:56,954 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - server_urls: 'http://example.com' (source: Environment Variables)
2023-07-07 07:26:56,954 [main] WARN co.elastic.apm.agent.configuration.StartupInfo - To enable all features and decrease startup time, please configure application_packages
2023-07-07 07:26:57,213 [main] INFO co.elastic.apm.agent.bci.ElasticApmAgent - Overriding IndyBootstrapDispatcher class's module to java.base module. This is required in J9 VMs.
2023-07-07 07:26:57,216 [main] INFO co.elastic.apm.agent.bci.ElasticApmAgent - Overriding IndyBootstrapDispatcher class's module to java.base module. This is required in J9 VMs.
2023-07-07 07:26:58,072 [main] INFO co.elastic.apm.agent.impl.ElasticApmTracer - Tracer switched to RUNNING state
2023-07-07 07:26:58,444 [elastic-apm-server-healthcheck] INFO co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is available: <!doctype html><html><head> <title>Example Domain</title> <meta charset="utf-8" /> <meta http-equiv="Content-type" content="text/html; charset=utf-8" /> <meta name="viewport" content="width=device-width, initial-scale=1" /> <style type="text/css"> body { background-color: #f0f0f2; margin: 0; padding: 0; font-family: -apple-system, system-ui, BlinkMacSystemFont, "Segoe UI", "Open Sans", "Helvetica Neue", Helvetica, Arial, sans-serif; } div { width: 600px; margin: 5em auto; padding: 2em; background-color: #fdfdff; border-radius: 0.5em; box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02); } a:link, a:visited { color: #38488f; text-decoration: none; } @media (max-width: 700px) { div { margin: 0 auto; width: auto; } } </style> </head><body><div> <h1>Example Domain</h1> <p>This domain is for use in illustrative examples in documents. You may use this domain in literature without prior coordination or asking for permission.</p> <p><a href="https://www.iana.org/domains/example">More information...</a></p></div></body></html>
2023-07-07 07:26:58,447 [elastic-apm-server-healthcheck] WARN co.elastic.apm.agent.report.ApmServerHealthChecker - Failed to parse version of APM server http://example.com/: Expecting '{' as object start. Found < at position: 1, following: `<`
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v3.1.1)
2023-07-07T07:26:59.057Z INFO 1 --- [ main] com.example.demo.DemoApplication : Starting DemoApplication v0.0.1-SNAPSHOT using Java 17.0.6 with PID 1 (/work/demo-0.0.1-SNAPSHOT.jar started by root in /)
2023-07-07T07:26:59.067Z INFO 1 --- [ main] com.example.demo.DemoApplication : No active profile set, falling back to 1 default profile: "default"
2023-07-07T07:27:00.136Z INFO 1 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)
2023-07-07T07:27:00.149Z INFO 1 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2023-07-07T07:27:00.149Z INFO 1 --- [ main] o.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/10.1.10]
2023-07-07T07:27:00.221Z INFO 1 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2023-07-07T07:27:00.222Z INFO 1 --- [ main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 1090 ms
2023-07-07T07:27:00.722Z INFO 1 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path ''
2023-07-07T07:27:00.735Z INFO 1 --- [ main] com.example.demo.DemoApplication : Started DemoApplication in 2.272 seconds (process running for 4.254)
2023-07-07 07:27:38,749 [elastic-apm-server-reporter] WARN co.elastic.apm.agent.report.AbstractIntakeApiHandler - Response body: <?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<title>404 - Not Found</title>
</head>
<body>
<h1>404 - Not Found</h1>
<script type="text/javascript" src="//obj.ac.bcon.ecdns.net/ec_tpm_bcon.js"></script>
</body>
</html>
Any idea if this is an issue with the agent or if something needs to be raised with openj9?