JVM crashes with APM agent 1.33 and JPMS enabled

Elastic Observability APM
1

Kibana version: 7.17.4

Elasticsearch version: 7.17.4

APM Server version: 7.17.4

APM Agent language and version: 1.33.0

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant): Agent is attached via -javaagent arg.
Application is based on spring boot 2.7.1 (with log4j config defined).
APM agent is using defaults (no explicit config defined for logging etc).
Environment is Linux OpenJDK 64-Bit Server VM Temurin-17.0.3+7,
JVM exits after about 2 seconds from startup, with the following exception logged to STDOUT.

java.lang.NullPointerException: Cannot invoke "co.elastic.apm.agent.log4j2.reformatting.Log4J2EcsReformattingHelper.getEcsOverridingFormatterFor(Object)" because "co.elastic.apm.agent.log4j2.reformatting.Log4j2AppenderGetLayoutAdvice.helper" is null
        at co.elastic.apm.agent.log4j2.reformatting.Log4j2AppenderGetLayoutAdvice.reformatLoggingEvent(Log4j2AppenderGetLayoutAdvice.java:50)
        at org.apache.logging.log4j.core@2.17.2/org.apache.logging.log4j.core.appender.AbstractAppender.getLayout(AbstractAppender.java:264)
        at org.apache.logging.log4j.core@2.17.2/org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.start(AbstractOutputStreamAppender.java:163)
        at org.apache.logging.log4j.core@2.17.2/org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:311)
        at org.apache.logging.log4j.core@2.17.2/org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:621)
        at org.apache.logging.log4j.core@2.17.2/org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:694)
        at org.apache.logging.log4j.core@2.17.2/org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:711)
        at org.apache.logging.log4j.core@2.17.2/org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
        at org.apache.logging.log4j.core@2.17.2/org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:155)
        at org.apache.logging.log4j.core@2.17.2/org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:47)
        at org.apache.logging.log4j@2.17.2/org.apache.logging.log4j.LogManager.getContext(LogManager.java:196)
        at org.apache.logging.log4j@2.17.2/org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:137)
        at co.elastic.apm.agent.logging.Log4jLoggerFactoryBridge.getContext(Log4jLoggerFactoryBridge.java:64)
        at org.apache.logging.log4j@2.17.2/org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:47)
        at co.elastic.apm.agent.logging.Log4jLoggerFactoryBridge.getLogger(Log4jLoggerFactoryBridge.java:37)
        at co.elastic.apm.agent.sdk.logging.LoggerFactory$1.run(LoggerFactory.java:68)
        at co.elastic.apm.agent.sdk.logging.LoggerFactory$1.run(LoggerFactory.java:65)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:318)
        at co.elastic.apm.agent.sdk.logging.LoggerFactory.getLoggerFromFactory(LoggerFactory.java:65)
        at co.elastic.apm.agent.sdk.logging.LoggerFactory.getLogger(LoggerFactory.java:45)
        at co.elastic.apm.agent.sdk.logging.LoggerFactory.getLogger(LoggerFactory.java:56)
        at co.elastic.apm.agent.log4j2.reformatting.Log4J2EcsReformattingHelper.<clinit>(Log4J2EcsReformattingHelper.java:49)
        at co.elastic.apm.agent.log4j2.reformatting.Log4j2AppenderGetLayoutAdvice.<clinit>(Log4j2AppenderGetLayoutAdvice.java:37)
        at java.base/jdk.internal.misc.Unsafe.ensureClassInitialized0(Native Method)
        at java.base/jdk.internal.misc.Unsafe.ensureClassInitialized(Unsafe.java:1155)
        at java.base/java.lang.invoke.DirectMethodHandle$EnsureInitialized.computeValue(DirectMethodHandle.java:377)
        at java.base/java.lang.invoke.DirectMethodHandle$EnsureInitialized.computeValue(DirectMethodHandle.java:374)
        at java.base/java.lang.ClassValue.getFromHashMap(ClassValue.java:228)
        at java.base/java.lang.ClassValue.getFromBackup(ClassValue.java:210)
        at java.base/java.lang.ClassValue.get(ClassValue.java:116)
        at java.base/java.lang.invoke.DirectMethodHandle.checkInitialized(DirectMethodHandle.java:400)
        at java.base/java.lang.invoke.DirectMethodHandle.ensureInitialized(DirectMethodHandle.java:388)
        at java.base/java.lang.invoke.DirectMethodHandle.internalMemberNameEnsureInit(DirectMethodHandle.java:338)
        at org.apache.logging.log4j.core@2.17.2/org.apache.logging.log4j.core.appender.AbstractAppender.getLayout(AbstractAppender.java:264)
        at org.apache.logging.log4j.core@2.17.2/org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.start(AbstractOutputStreamAppender.java:163)
        at org.apache.logging.log4j.core@2.17.2/org.apache.logging.log4j.core.config.AbstractConfiguration.setToDefault(AbstractConfiguration.java:731)
        at org.apache.logging.log4j.core@2.17.2/org.apache.logging.log4j.core.config.DefaultConfiguration.<init>(DefaultConfiguration.java:47)
        at org.apache.logging.log4j.core@2.17.2/org.apache.logging.log4j.core.LoggerContext.<init>(LoggerContext.java:84)
        at org.apache.logging.log4j.core@2.17.2/org.apache.logging.log4j.core.selector.ClassLoaderContextSelector.createContext(ClassLoaderContextSelector.java:254)
        at org.apache.logging.log4j.core@2.17.2/org.apache.logging.log4j.core.selector.ClassLoaderContextSelector.locateContext(ClassLoaderContextSelector.java:218)
        at org.apache.logging.log4j.core@2.17.2/org.apache.logging.log4j.core.selector.ClassLoaderContextSelector.getContext(ClassLoaderContextSelector.java:136)
        at org.apache.logging.log4j.core@2.17.2/org.apache.logging.log4j.core.selector.ClassLoaderContextSelector.getContext(ClassLoaderContextSelector.java:123)
        at org.apache.logging.log4j.core@2.17.2/org.apache.logging.log4j.core.selector.ClassLoaderContextSelector.getContext(ClassLoaderContextSelector.java:117)
        at org.apache.logging.log4j.core@2.17.2/org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:150)
        at org.apache.logging.log4j.core@2.17.2/org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:47)
        at org.apache.logging.log4j@2.17.2/org.apache.logging.log4j.LogManager.getContext(LogManager.java:196)
        at org.apache.logging.log4j@2.17.2/org.apache.logging.log4j.LogManager.getLogger(LogManager.java:599)
        at org.apache.logging.log4j@2.17.2/org.apache.logging.log4j.LogManager.getLogger(LogManager.java:585)
        at com.mycompany.servicebase.spring@50.1.0/com.mycompany.servicebase.spring.ToolboxService.<clinit>(ToolboxService.java:38)

Steps to reproduce:
1.
2.
3.

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

2

Hey @emmettl , welcome to our forum and thanks for reporting!

Please share the entire stack trace. An NPE during instrumentation should not crash the app, so it would be interesting to see everything. From the part you shared, it looked as if it's on its way to stack overflow. Our safeguards of not instrumenting our own logging may be escaped in this case.
If the error/log is too big, you can share through https://gist.github.com/

3

Ahh, it may escape our checks because they rely on whether the class loader loading the logger is an agent CL or not. Maybe it is not identified in this case because JPMS does something differently...
The full log may assist with analysis

4

I've updated the full stack trace. Hope that helps.

5

Hi @emmettl thanks for this report. The agent currently fails when the application uses JPMS and they both (agent and application) have a common dependency. The issue has been identified but not yet prioritized.

6

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.