APM Java agent v1.27.0+ instrumention errors on modulepath

duponter · December 24, 2021, 8:23am

Currently, APM Java agent v1.26.0 is attached successfully to a Spring Boot 2.6 application (JDK 17).
This application has enabled Java modules and is therefore running on the modulepath.

Although the APM Java Agent is not impacted to the Log4J vulnerability, we want to upgrade to the latest version (v1.28.3 atm). During start up, following exceptions are thrown abundantly:

c.e.a.a.b.b.ErrorLoggingListener         : Error on transformation org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler
    java.lang.AbstractMethodError: Receiver class co.elastic.apm.agent.bci.bytebuddy.MinimumClassFileVersionValidator does not define or inherit an implementation of the resolved method 'abstract net.bytebuddy.jar.asm.ClassVisitor wrap(net.bytebuddy.description.type.TypeDescription, net.bytebuddy.jar.asm.ClassVisitor, net.bytebuddy.implementation.Implementation$Context, net.bytebuddy.pool.TypePool, net.bytebuddy.description.field.FieldList, net.bytebuddy.description.method.MethodList, int, int)' of interface net.bytebuddy.asm.AsmVisitorWrapper.
        at net.bytebuddy@1.11.22/net.bytebuddy.asm.AsmVisitorWrapper$Compound.wrap(AsmVisitorWrapper.java:743) ~[byte-buddy-1.11.22.jar:na]
        at net.bytebuddy@1.11.22/net.bytebuddy.dynamic.scaffold.TypeWriter$Default$ForInlining$WithFullProcessing$RedefinitionClassVisitor.visit(TypeWriter.java:4867) ~[byte-buddy-1.11.22.jar:na]
        at net.bytebuddy@1.11.22/net.bytebuddy.jar.asm.ClassReader.accept(ClassReader.java:569) ~[byte-buddy-1.11.22.jar:na]
        at net.bytebuddy@1.11.22/net.bytebuddy.jar.asm.ClassReader.accept(ClassReader.java:424) ~[byte-buddy-1.11.22.jar:na]
        at net.bytebuddy@1.11.22/net.bytebuddy.dynamic.scaffold.TypeWriter$Default$ForInlining.create(TypeWriter.java:3918) ~[byte-buddy-1.11.22.jar:na]
        at net.bytebuddy@1.11.22/net.bytebuddy.dynamic.scaffold.TypeWriter$Default.make(TypeWriter.java:2192) ~[byte-buddy-1.11.22.jar:na]
        at net.bytebuddy@1.11.22/net.bytebuddy.dynamic.scaffold.inline.RedefinitionDynamicTypeBuilder.make(RedefinitionDynamicTypeBuilder.java:224) ~[byte-buddy-1.11.22.jar:na]
        at net.bytebuddy@1.11.22/net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doTransform(AgentBuilder.java:11362) ~[byte-buddy-1.11.22.jar:na]
        at net.bytebuddy@1.11.22/net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:11298) ~[byte-buddy-1.11.22.jar:na]
        at net.bytebuddy@1.11.22/net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.access$1700(AgentBuilder.java:11016) ~[byte-buddy-1.11.22.jar:na]
        at net.bytebuddy@1.11.22/net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:11763) ~[byte-buddy-1.11.22.jar:na]
        at net.bytebuddy@1.11.22/net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:11702) ~[byte-buddy-1.11.22.jar:na]
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:399) ~[na:na]
        at net.bytebuddy@1.11.22/net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doPrivileged(AgentBuilder.java) ~[byte-buddy-1.11.22.jar:na]
        at net.bytebuddy@1.11.22/net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:11240) ~[byte-buddy-1.11.22.jar:na]
        at net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$ByteBuddy$ModuleSupport.transform(Unknown Source) ~[na:na]
        at java.instrument/sun.instrument.TransformerManager.transform(TransformerManager.java:188) ~[na:na]
        at java.instrument/sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:541) ~[na:na]
        at java.base/java.lang.ClassLoader.defineClass1(Native Method) ~[na:na]
        at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1012) ~[na:na]
        at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1090) ~[na:na]
        at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:182) ~[na:na]
        at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:821) ~[na:na]
        at java.base/jdk.internal.loader.BuiltinClassLoader.findClassInModuleOrNull(BuiltinClassLoader.java:741) ~[na:na]
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:665) ~[na:na]
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:639) ~[na:na]
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188) ~[na:na]
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520) ~[na:na]

When reverting to the v1.26 version, the agent attaches normally.
When running the same application on the classpath, the agent v1.28.3 attaches normally.

After more research, we suspect the error appears after following change has been applied:

github.com/elastic/apm-agent-java

Shade byte-buddy in attacher jar

elastic:master ← SylvainJuge:bytebuddy-shade-attacher

opened 12:34PM - 10 Sep 21 UTC

SylvainJuge

+847 -38

## What does this PR do? Since version 1.23.0, the attacher jar does not embe…ds bytebuddy dependency anymore. This becomes an issue when a different version of bytebuddy is on the application classpath (for example through an hibernate dependency) as it makes the attach fail with a similar stack trace: ``` Exception in thread "main" java.lang.reflect.InvocationTargetException at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) at org.springframework.boot.loader.Launcher.launch(Launcher.java:50) at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51) Caused by: java.lang.NoClassDefFoundError: net/bytebuddy/agent/ByteBuddyAgent$AttachmentProvider$ForEmulatedAttachment at co.elastic.apm.attach.ElasticAttachmentProvider.init(ElasticAttachmentProvider.java:39) at co.elastic.apm.attach.ElasticAttachmentProvider.get(ElasticAttachmentProvider.java:60) at co.elastic.apm.attach.ElasticApmAttacher.attach(ElasticApmAttacher.java:162) at co.elastic.apm.attach.ElasticApmAttacher.attach(ElasticApmAttacher.java:148) at co.elastic.apm.attach.ElasticApmAttacher.attach(ElasticApmAttacher.java:110) at co.elastic.apm.attach.ElasticApmAttacher.attach(ElasticApmAttacher.java:63) at org.springframework.samples.petclinic.PetClinicApplication.main(PetClinicApplication.java:34) ... 8 more Caused by: java.lang.ClassNotFoundException: net.bytebuddy.agent.ByteBuddyAgent$AttachmentProvider$ForEmulatedAttachment at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589) at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:93) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) ... 15 morev ``` This previous shading change has been introduced in version 1.23.0 through [this commit](https://github.com/elastic/apm-agent-java/pull/1667/commits/d6a262f02f7e07b8be55750a59636fa7c60818e1) which is part of [PR #1667](https://github.com/elastic/apm-agent-java/pull/1667) Since `attacher-cli` had a few minor dependencies on bytebuddy and it was provided through a transitive dependency of `attacher` which now relocates bytebuddy, it was necessary to move few parts to `attacher` for convenience and to allow removing dependencies from `attacher-cli` to bytebuddy. ## Checklist - [x] This is a bugfix - [x] I have updated [CHANGELOG.asciidoc](https://github.com/elastic/apm-agent-java/blob/master/CHANGELOG.asciidoc) - [x] I have added tests that would fail without this fix - [x] test on Windows

We could reproduce this error when generating a new Spring Boot application (with only Spring Data Jpa and Spring MVC) and converting it to the module path.

For now, we are sticking to the latest v1.26.x Agent.

Jack_Shirazi · December 24, 2021, 10:07am

Thanks. 1.26.1 is available as you note, exactly because of the large architectural change that you see in 1.27. What would be ideal for us is if you can provide an isolated example which triggers that bug.

duponter · December 24, 2021, 11:53am

As I mentioned in the topic, we could reproduce the bug with a newly generated Spring Boot application.

The following link refers to an apm-modules.zip file which contains a single module Maven project with a simple Spring Boot 2.6 application.

It contains a readme.md with instructions to run and trigger the bug.
It does not include the APM agent jars (to keep the zip as small as possible). You’ll have to add them manually in the project’s root directory.
Hopefully, it helps you any further.

Thanks for looking into this

Jack_Shirazi · December 24, 2021, 3:10pm

Thank you, that's very useful. It looks like it'll take us some time (especially given holiday period) to resolve this

Sylvain_Juge · January 4, 2022, 9:04am

We also have a github issue (Fix modularized applications having a dependency version conflict · Issue #2373 · elastic/apm-agent-java · GitHub) to keep track of this bug. You can subscribe to it to get notified when we are making progress on it.

system · January 25, 2022, 5:05am

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.