Hello,
The APM agent has caused a JVM crash on a production Tomcat system.
Kibana version: 9.4.0
Elasticsearch version: 9.4.0
APM Server version: 9.4.0
APM Agent language and version: APM Java agent version 1.55.6 (latest)
Is there anything special in your setup? Nothing special. I created custom ingest pipelines to reroute APM data to specific data_streams.
JVM Error is:
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f064da173ee, pid=2046380, tid=2047085
#
# JRE version: OpenJDK Runtime Environment (Red_Hat-21.0.3.0.9-1) (21.0.3+9) (build 21.0.3+9-LTS)
# Java VM: OpenJDK 64-Bit Server VM (Red_Hat-21.0.3.0.9-1) (21.0.3+9-LTS, mixed mode, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# J 92993 c2 co.elastic.apm.agent.loginstr.reformatting.AbstractEcsReformattingHelper.onAppendEnter(Ljava/lang/Object;)Z (126 bytes) @ 0x00007f064da173ee [0x00007f064da16f80+0x000000000000046e]
#
# Core dump will be written. Default location: Core dumps may be processed with "/opt/dynatrace/oneagent/agent/rdp -p %p -P %P -c %c -e %e -s %s -a %P %u %g %s %t %c %h" (or dumping to /aplicaciones-api/apache-tomcat-10.1.31/bin/core.2046380)
#
# If you would like to submit a bug report, please visit:
# https://access.redhat.com/support/cases/
#
--------------- S U M M A R Y ------------
Command Line: -Djava.util.logging.config.file=/aplicaciones/tomcat-api/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -XX:MaxGCPauseMillis=500 -XX:+DisableExplicitGC -Xms128m -Xmx5120m -XX:+UseG1GC -XX:ParallelGCThreads=6 -XX:+UseStringDeduplication -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/aplicaciones/tomcat-api/logs -XX:+ForceTimeHighResolution -Dfile.encoding=iso-8859-1 -Dcom.sun.jndi.ldap.connect.pool.prefsize=5 -Dcom.sun.jndi.ldap.connect.pool.timeout=30000 -Dnet.sf.ehcache.skipUpdateCheck=true -Djava.file.encoding=iso-8859-1 -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9897 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.host=HOST -javaagent:/opt/Elastic/apm/elastic-apm-agent-1.55.6.jar -Dcatalina.base=/aplicaciones/tomcat-api -Dcatalina.home=/aplicaciones/tomcat-api -Djava.io.tmpdir=/aplicaciones/tomcat-api/temp -agentpath:/opt/dynatrace/oneagent/agent/bin/1.307.61.20250226-155505/linux-x86-64/liboneagentjava.so=loglevelcon=none,datastorage=/opt/dynatrace/datastorage,logdir=/var/log/dynatrace/oneagent org.apache.catalina.startup.Bootstrap start
Host: Intel(R) Xeon(R) Gold 6240R CPU @ 2.40GHz, 6 cores, 29G, Red Hat Enterprise Linux release 9.7 (Plow)
Time: Tue May 26 16:12:49 2026 CEST elapsed time: 553140.319412 seconds (6d 9h 39m 0s)
--------------- T H R E A D ---------------
Current thread (0x0000565328ecb650): JavaThread "http-nio-IP-PORT-exec-19" daemon [_thread_in_Java, id=2047085, stack(0x00007f066a5e7000,0x00007f066a6e7000) (1024K)]
Stack: [0x00007f066a5e7000,0x00007f066a6e7000], sp=0x00007f066a6e4970, free space=1014k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
J 92993 c2 co.elastic.apm.agent.loginstr.reformatting.AbstractEcsReformattingHelper.onAppendEnter(Ljava/lang/Object;)Z (126 bytes) @ 0x00007f064da173ee [0x00007f064da16f80+0x000000000000046e]
j co.elastic.apm.agent.jul.reformatting.JulConsoleHandlerPublishAdvice.initializeReformatting(Ljava/util/logging/ConsoleHandler;)Z+4
J 333909 c2 java.util.logging.Logger.log(Ljava/util/logging/LogRecord;)V java.logging@21.0.3 (153 bytes) @ 0x00007f0652febf08 [0x00007f0652febde0+0x0000000000000128]
J 80103 c1 java.util.logging.Logger.doLog(Ljava/util/logging/LogRecord;)V java.logging@21.0.3 (50 bytes) @ 0x00007f0646a8cbf4 [0x00007f0646a8c7e0+0x0000000000000414]
J 165298 c1 java.util.logging.Logger.logp(Ljava/util/logging/Level;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/Throwable;)V java.logging@21.0.3 (47 bytes) @ 0x00007f064a087a84 [0x00007f064a087500+0x0000000000000584]
J 133115 c1 org.apache.juli.logging.DirectJDKLog.log(Ljava/util/logging/Level;Ljava/lang/String;Ljava/lang/Throwable;)V (102 bytes) @ 0x00007f0649bceabc [0x00007f0649bce2c0+0x00000000000007fc]
J 97846 c2 org.apache.catalina.mbeans.MBeanDumper.dumpBeans(Ljavax/management/MBeanServer;Ljava/util/Set;)Ljava/lang/String; (758 bytes) @ 0x00007f064e833160 [0x00007f064e82dd60+0x0000000000005400]
J 174678 c2 org.apache.catalina.manager.JMXProxyServlet.doGet(Ljakarta/servlet/http/HttpServletRequest;Ljakarta/servlet/http/HttpServletResponse;)V (913 bytes) @ 0x00007f06503f1e54 [0x00007f06503f0a40+0x0000000000001414]
J 272536 c2 jakarta.servlet.http.HttpServlet.service(Ljakarta/servlet/http/HttpServletRequest;Ljakarta/servlet/http/HttpServletResponse;)V (271 bytes) @ 0x00007f06520197c4 [0x00007f0652019620+0x00000000000001a4]
J 127667 c2 jakarta.servlet.http.HttpServlet.service(Ljakarta/servlet/ServletRequest;Ljakarta/servlet/ServletResponse;)V (122 bytes) @ 0x00007f064e70b378 [0x00007f064e70b140+0x0000000000000238]
J 95237 c2 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Ljakarta/servlet/ServletRequest;Ljakarta/servlet/ServletResponse;)V (386 bytes) @ 0x00007f064dd22a74 [0x00007f064dd22900+0x0000000000000174]
J 127845 c2 org.apache.tomcat.websocket.server.WsFilter.doFilter(Ljakarta/servlet/ServletRequest;Ljakarta/servlet/ServletResponse;Ljakarta/servlet/FilterChain;)V (614 bytes) @ 0x00007f064f5ac520 [0x00007f064f5abd40+0x00000000000007e0]
J 95237 c2 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Ljakarta/servlet/ServletRequest;Ljakarta/servlet/ServletResponse;)V (386 bytes) @ 0x00007f064dd22b48 [0x00007f064dd22900+0x0000000000000248]
J 173851 c2 org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(Ljakarta/servlet/ServletRequest;Ljakarta/servlet/ServletResponse;Ljakarta/servlet/FilterChain;)V (614 bytes) @ 0x00007f065059d21c [0x00007f065059c900+0x000000000000091c]
J 95237 c2 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Ljakarta/servlet/ServletRequest;Ljakarta/servlet/ServletResponse;)V (386 bytes) @ 0x00007f064dd22b48 [0x00007f064dd22900+0x0000000000000248]
J 180539 c2 org.apache.catalina.core.StandardWrapperValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V (2681 bytes) @ 0x00007f065062cc50 [0x00007f065062c6a0+0x00000000000005b0]
J 263362 c2 org.apache.catalina.core.StandardContextValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V (175 bytes) @ 0x00007f064ebd2ba0 [0x00007f064ebd2200+0x00000000000009a0]
J 108080 c2 org.apache.catalina.authenticator.AuthenticatorBase.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V (820 bytes) @ 0x00007f064ef33b30 [0x00007f064ef333c0+0x0000000000000770]
J 263161 c2 org.apache.catalina.core.StandardHostValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V (405 bytes) @ 0x00007f064fda0c30 [0x00007f064fd9fba0+0x0000000000001090]
J 261130 c2 org.apache.catalina.core.StandardEngineValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V (65 bytes) @ 0x00007f065120a548 [0x00007f0651209f20+0x0000000000000628]
J 263304 c2 org.apache.catalina.connector.CoyoteAdapter.service(Lorg/apache/coyote/Request;Lorg/apache/coyote/Response;)V (893 bytes) @ 0x00007f064f3100a4 [0x00007f064f30fd80+0x0000000000000324]
J 291047 c2 org.apache.coyote.http11.Http11Processor.service(Lorg/apache/tomcat/util/net/SocketWrapperBase;)Lorg/apache/tomcat/util/net/AbstractEndpoint$Handler$SocketState; (1252 bytes) @ 0x00007f0652191d48 [0x00007f0652191900+0x0000000000000448]
J 292372 c2 org.apache.coyote.AbstractProtocol$ConnectionHandler.process(Lorg/apache/tomcat/util/net/SocketWrapperBase;Lorg/apache/tomcat/util/net/SocketEvent;)Lorg/apache/tomcat/util/net/AbstractEndpoint$Handler$SocketState; (1288 bytes) @ 0x00007f065224b5a0 [0x00007f065224b1a0+0x0000000000000400]
J 104022 c2 org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun()V (600 bytes) @ 0x00007f064df025f8 [0x00007f064df02520+0x00000000000000d8]
J 103766 c2 org.apache.tomcat.util.net.SocketProcessorBase.run()V (264 bytes) @ 0x00007f064df68f60 [0x00007f064df68dc0+0x00000000000001a0]
j org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(Lorg/apache/tomcat/util/threads/ThreadPoolExecutor$Worker;)V+92
j org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run()V+100
j org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run()V+105
J 67137 c1 java.lang.Thread.runWith(Ljava/lang/Object;Ljava/lang/Runnable;)V java.base@21.0.3 (15 bytes) @ 0x00007f06479fd884 [0x00007f06479fd740+0x0000000000000144]
j java.lang.Thread.run()V+19 java.base@21.0.3
v ~StubRoutines::call_stub 0x00007f064c537cc6
V [libjvm.so+0x926925] JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, JavaThread*)+0x315 (javaCalls.cpp:415)
V [libjvm.so+0x92830b] JavaCalls::call_virtual(JavaValue*, Handle, Klass*, Symbol*, Symbol*, JavaThread*)+0x1cb (javaCalls.cpp:329)
V [libjvm.so+0xa03b09] thread_entry(JavaThread*, JavaThread*)+0x99 (jvm.cpp:2918)
V [libjvm.so+0x93eb9f] JavaThread::thread_main_inner()+0x1ef (javaThread.cpp:719)
V [libjvm.so+0xfaf527] Thread::call_run()+0xb7 (thread.cpp:218)
V [libjvm.so+0xd2ee8a] thread_native_entry(Thread*)+0xda (os_linux.cpp:783)
siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000427418
Registers:
RAX=0x00000006cef81c88, RBX=0x0000000000427418, RCX=0x0000000000000016, RDX=0x00000006cef88310
RSP=0x00007f066a6e4970, RBP=0x00000006db729160, RSI=0x00000006ddece540, RDI=0x0000000000000000
R8 =0x0000000000084e83, R9 =0x00000000d9df1062, R10=0x00000000000001b8, R11=0x00007f05e065beb0
R12=0x0000000000000000, R13=0x00007f068d277c0e, R14=0x00000006cef81c88, R15=0x0000565328ecb650
RIP=0x00007f064da173ee, EFLAGS=0x0000000000010246, CSGSFS=0x002b000000000033, ERR=0x0000000000000004
TRAPNO=0x000000000000000e
Register to memory mapping:
RAX=0x00000006cef81c88 is an oop: co.elastic.apm.agent.weakconcurrent.CachedLookupKey
{0x00000006cef81c88} - klass: 'co/elastic/apm/agent/weakconcurrent/CachedLookupKey'
- ---- fields (total size 3 words):
- private 'hashCode' 'I' @12 0 (0x00000000)
- private 'key' 'Ljava/lang/Object;' @16
[error occurred during error reporting (printing register info), id 0xb, SIGSEGV (0xb) at pc=0x00007f06a8914d43]
RBX=4355096 is a compressed pointer to class: 0x0000000800427418
java.util.logging.ConsoleHandler {0x0000000800427418}
It appears to be due to co.elastic.apm.agent.loginstr.reformatting.AbstractEcsReformattingHelper.onAppendEnter but we have log-ecs-reformatting disabled.
Could this be related to:
- Java 21 compatibility with agent 1.55.6?
- Log reformatting instrumentation?
- Interaction with another Java agent?
- Is this a known issue in this version of the APM agent?
- Would disabling log reformatting (
disable_instrumentations=log-reformatting) be recommended as a workaround?