Trying external-plugin-test on "real setup" fails with "Cannot resolve type description for co.elastic.apm.agent.bci.classloading.LookupExposer"

Hey there,

when I try out the external-plugin-test from integration-tests, it fails with "Cannot resolve type description for co.elastic.apm.agent.bci.classloading.LookupExposer".
So probably something is missing / not loaded correctly - maybe I am doing something wrong. Hopefully, someone can point me in the right direction?

Setup:

  • The main does just the attach and then calls traceMe().
  • TestClass with a traceMe() method is the method which I try to instrument (i.e. I set the System properties).
  • I did a small change to the original plugin: I print out "Called trace me" in the traceMe(). And this is printed, just after and just before the "Cannot resolve type description for co.elastic.apm.agent.bci.classloading.LookupExposer".

Versions:
APM-Agent Version: 1.19.0 (also tried briefly with 1.20.0, but failed there as well).
Attachment: Via co.elastic.apm.attach.ElasticApmAttacher.attach(); . Tried the javaagent as well, no difference.

Expected:

  • To be working like in the test? :slight_smile:

Log (short):

Instrumented class: externalplugin.TestClass
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
2021-01-13 00:57:01,795 [Attach Listener] INFO  co.elastic.apm.agent.util.JmxUtils - Found JVM-specific OperatingSystemMXBean interface: com.sun.management.OperatingSystemMXBean
2021-01-13 00:57:01,820 [Attach Listener] INFO  co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM 1.19.0 as externalplugin-test on Java 14.0.1 Runtime version: 14.0.1+7 VM version: 14.0.1+7 (AdoptOpenJDK) Mac OS X 10.15.7
2021-01-13 00:57:01,821 [Attach Listener] INFO  co.elastic.apm.agent.configuration.StartupInfo - VM Arguments: [-Delastic.apm.plugin.instrumented_class=externalplugin.TestClass, -Delastic.apm.plugin.instrumented_method=traceMe, -Dfile.encoding=UTF-8, -Duser.country=DE, -Duser.language=en, -Duser.variant]
2021-01-13 00:57:01,826 [Attach Listener] INFO  co.elastic.apm.agent.bci.ElasticApmAgent - Loading plugin external-plugin-test-1.19.0.jar
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
2021-01-13 00:57:03,138 [Attach Listener] INFO  co.elastic.apm.agent.impl.ElasticApmTracer - Tracer switched to RUNNING state
2021-01-13 00:57:03,163 [main] ERROR co.elastic.apm.agent.bci.IndyBootstrap - Cannot resolve type description for co.elastic.apm.agent.bci.classloading.LookupExposer
java.lang.IllegalStateException: Cannot resolve type description for co.elastic.apm.agent.bci.classloading.LookupExposer
	at co.elastic.apm.agent.shaded.bytebuddy.pool.TypePool$Resolution$Illegal.resolve(TypePool.java:157) ~[?:?]
	at co.elastic.apm.agent.shaded.bytebuddy.pool.TypePool$Default$WithLazyResolution$LazyTypeDescription.delegate(TypePool.java:912) ~[?:?]
	at co.elastic.apm.agent.shaded.bytebuddy.description.type.TypeDescription$AbstractBase$OfSimpleType$WithDelegation.getDeclaredAnnotations(TypeDescription.java:8348) ~[?:?]
	at co.elastic.apm.agent.shaded.bytebuddy.matcher.DeclaringAnnotationMatcher.matches(DeclaringAnnotationMatcher.java:48) ~[?:?]
	at co.elastic.apm.agent.shaded.bytebuddy.matcher.DeclaringAnnotationMatcher.matches(DeclaringAnnotationMatcher.java:27) ~[?:?]
	at co.elastic.apm.agent.shaded.bytebuddy.matcher.ElementMatcher$Junction$Disjunction.matches(ElementMatcher.java:160) ~[?:?]
	at co.elastic.apm.agent.bci.IndyPluginClassLoaderFactory.getOrCreatePluginClassLoader(IndyPluginClassLoaderFactory.java:73) ~[?:?]
	at co.elastic.apm.agent.bci.IndyBootstrap.bootstrap(IndyBootstrap.java:317) [?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:564) ~[?:?]
	at java.lang.IndyBootstrapDispatcher.bootstrap(IndyBootstrapDispatcher.java:31) [?:?]
	at java.lang.invoke.BootstrapMethodInvoker.invoke(BootstrapMethodInvoker.java:138) [?:?]
	at java.lang.invoke.CallSite.makeSite(CallSite.java:315) [?:?]
	at java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:259) [?:?]
	at java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:249) [?:?]
	at externalplugin.TestClass.traceMe(TestClass.java:9) [main/:?]
	at externalplugin.TestClass.<init>(TestClass.java:5) [main/:?]
	at externalplugin.Main.main(Main.java:20) [main/:?]
Called trace me
2021-01-13 00:57:03,208 [main] ERROR co.elastic.apm.agent.bci.IndyBootstrap - Cannot resolve type description for co.elastic.apm.agent.bci.classloading.LookupExposer
java.lang.IllegalStateException: Cannot resolve type description for co.elastic.apm.agent.bci.classloading.LookupExposer
	at co.elastic.apm.agent.shaded.bytebuddy.pool.TypePool$Resolution$Illegal.resolve(TypePool.java:157) ~[?:?]
	at co.elastic.apm.agent.shaded.bytebuddy.pool.TypePool$Default$WithLazyResolution$LazyTypeDescription.delegate(TypePool.java:912) ~[?:?]
	at co.elastic.apm.agent.shaded.bytebuddy.description.type.TypeDescription$AbstractBase$OfSimpleType$WithDelegation.getDeclaredAnnotations(TypeDescription.java:8348) ~[?:?]
	at co.elastic.apm.agent.shaded.bytebuddy.matcher.DeclaringAnnotationMatcher.matches(DeclaringAnnotationMatcher.java:48) ~[?:?]
	at co.elastic.apm.agent.shaded.bytebuddy.matcher.DeclaringAnnotationMatcher.matches(DeclaringAnnotationMatcher.java:27) ~[?:?]
	at co.elastic.apm.agent.shaded.bytebuddy.matcher.ElementMatcher$Junction$Disjunction.matches(ElementMatcher.java:160) ~[?:?]
	at co.elastic.apm.agent.bci.IndyPluginClassLoaderFactory.getOrCreatePluginClassLoader(IndyPluginClassLoaderFactory.java:73) ~[?:?]
	at co.elastic.apm.agent.bci.IndyBootstrap.bootstrap(IndyBootstrap.java:317) [?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:564) ~[?:?]
	at java.lang.IndyBootstrapDispatcher.bootstrap(IndyBootstrapDispatcher.java:31) [?:?]
	at java.lang.invoke.BootstrapMethodInvoker.invoke(BootstrapMethodInvoker.java:138) [?:?]
	at java.lang.invoke.CallSite.makeSite(CallSite.java:315) [?:?]
	at java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:259) [?:?]
	at java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:249) [?:?]
	at externalplugin.TestClass.traceMe(TestClass.java:10) [main/:?]
	at externalplugin.TestClass.<init>(TestClass.java:5) [main/:?]
	at externalplugin.Main.main(Main.java:20) [main/:?]
2021-01-13 00:57:03,216 [elastic-apm-init-instrumentation-shutdown-hook] INFO  co.elastic.apm.agent.impl.ElasticApmTracer - Tracer switched to STOPPED state

BUILD SUCCESSFUL in 3s
3 actionable tasks: 2 executed, 1 up-to-date
00:57:04: Task execution finished 'Main.main()'

Soo, digging a little bit deeper, it is probably due to missing to add the LookupExporter to some classloader. But I have no idea where and how.

That's because when I remove the LookupExporter introducing commit, 98707982806d720666c57e60f87c54de46220e93, everything works like expected.

Here is a project with the minimal test, where switching between the official libs and the libs without the bootstrap lookup fix can be used to see how the behaviour for external plugins changed with this commit (see build.gradle). See Main.java on how to run (there are System properties to be set).
Git repo for this https://github.com/mputz86/elastic-apm-external-plugin-test .

A workaround, which supports the assumption, that LookupExporter must be put somewhere else/loaded differently: Add the co.elastic.apm.agent.bci.classloading.LookupExposer to the external plugin sources (since it is an independent class anyway).

Hi and welcome to the APM forum!
Thanks for reporting. I could easily reproduce the issue using the provided project, so thanks for that as well! It will ease the analysis process.

@mputz86 unfortunately, it may take a while before we can prioritize looking into this. In order for us not to lose sight of this, please open a bug report on our GitHub repo with all the relevant details.