Custom plugin for java agent

Elastic Observability APM
1

Hello! I'm playing with apm java agent v1.4.0 and I'm very excited of it! Great work!
I chose attach artifact as most appropriate for my setup.
I use some very specific framework in enterprise env.

So, I would like to use official release of agent and code a custom plugin.
I see that agent uses java SPI to look for plugins. Also I see that when I use attach mechanism and obtain ElasticApmAgent class classloader I get null - so guess it is loaded with bootstrap classloader (?). While my plugin is loaded with system classloader. In this case service loader can't find my plugin implementation.

So, maybe I'm doing wrong - is there a way to implement a plugin outside of the official agent artifact?

2

Hi and thanks for the great feedback! :slight_smile:

Currently all plugins are indeed loaded from the bootstrap CL, however they can use classes loaded from any other CLs as well, as you may see in existing plugins we have. Why not build and package your plugin with the agent? Here is a blog post published just a couple of days ago that can guide you through. You should also be familiar with this page for technical guidelines on building, testing etc.

We can think of a way to load the plugin from other classloaders, but then we will have to make sure they are loaded properly before the classes you want to instrument. Using the existing agent mechanism can guarantee that.

We will be happy to hear more on what you are trying to do, and maybe provide more relevant assistance.

I hope this helps,
Eyal.

3

Hi! Thank you for the information! The post is great and super useful!

Yes, I can give you some more details about my case.
My env:

  • WebSphere ND 8.5.5.12 (I know it is not in list of supported tech)
  • Oracle 12c
  • the app is a bpmn platform - lets call it just platform
  • all the software and work activity is in company's closed network segment without internet connection

I've chosen elastic apm attach artifact for 2 reasons:

  1. would like to avoid app server java agent configuration
  2. facing some issues with OSGi and class loading (NoClassDefFoundErrors), when java agent is configured in app server.

Fortunately, everything works great if agent attaches to jvm after server startup during kinda platform startup task. So, at the moment agent attaches - app server is started, platform is started, a lot of classes apm agent instruments are loaded (for example, jdbc and stuff).

At this point everything is cool!

So, then I have some thoughts about custom build of agent with my plugin.
I see some disadvantages of having such a fork. I think, the main is codebase sync.
With the official release I can rely on your official release cycle, bugfixes, tests, benchmarks, improvements, and I hope, I could ask elastic-java-apm-agent-team for support here ) and you could reproduce some case using the official build. I'm happy with all of that )

On the other hand, I have my plugin with it's release cycle, tests, coverage, benchmarks and other stuff.

That is why I think it would be great feature to have an official build of agent and custom plugin alongside.

I'm not deep into classloading stuff, maybe I offer wrong things, but I have 2 probable options:

  • user could specify which classloader should attacher use to load agent jar
  • user could specify which classloader should ServiceLoader use to find ElasticApmInstrumentation implementations

The default behavior should stay as it is now.

4

While we wouldn't want to change the classloader that loads the agent (at least some minimal parts of it), as it means we would not be able to instrument core Java stuff, allowing to load plugins from different classloaders may be a valid option.
I created this PR that allows loading Instrumentation service providers from the system CL and its parents (platform/ext CL and bootstrap CL). My comment contain a snapshot of the attach jar for testing. Test it first and see if it works out. In the mean time we can consider if it makes sense for us to allow that in general or in some other way.

5

Hey! Thank you for response! But it doesn't work for me :confused:

I printed classloader hierarchy in place there I invoke attach() method for different CL.

  • system CL [ ClassLoader.getSystemClassLoader() ]:
    -- sun.misc.Launcher$AppClassLoader@5063017f
    -- sun.misc.Launcher$ExtClassLoader@b1526f5d

  • current class - class there I call co.elastic.apm.attach.ElasticApmAttacher.attach() [ getClass().getClassLoader() ]:
    -- platform-classloader@9e3cc701
    -- platform-classloader@1e4cfb34
    -- platform-classloader@e4dc9917
    -- platform-classloader@5f6bc538
    -- platform-classloader@e54b73f2
    -- com.ibm.ws.classloader.CompoundClassLoader@e39311fe[app:my_app]
    -- com.ibm.ws.classloader.ProtectionClassLoader@17cd07ee
    -- com.ibm.ws.bootstrap.ExtClassLoader@2dc2842
    -- org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader@69a89567
    -- sun.misc.Launcher$AppClassLoader@5063017f
    -- sun.misc.Launcher$ExtClassLoader@b1526f5d

  • thread CL [ Thread.currentThread().getContextClassLoader() ]:
    -- platform-classloader@e4dc9917
    -- platform-classloader@5f6bc538
    -- platform-classloader@e54b73f2
    -- com.ibm.ws.classloader.CompoundClassLoader@e39311fe[app:my_app]
    -- com.ibm.ws.classloader.ProtectionClassLoader@17cd07ee
    -- com.ibm.ws.bootstrap.ExtClassLoader@2dc2842
    -- org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader@69a89567
    -- sun.misc.Launcher$AppClassLoader@5063017f
    -- sun.misc.Launcher$ExtClassLoader@b1526f5d

  • custom apm plugin instumentation impl class CL [ co.elastic.apm.agent.custom.CustomInstrumentation.class.getClassLoader() ]:
    -- platform-classloader@e4dc9917 <-----
    -- platform-classloader@5f6bc538
    -- platform-classloader@e54b73f2
    -- com.ibm.ws.classloader.CompoundClassLoader@e39311fe[app:my_app]
    -- com.ibm.ws.classloader.ProtectionClassLoader@17cd07ee
    -- com.ibm.ws.bootstrap.ExtClassLoader@2dc2842
    -- org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader@69a89567
    -- sun.misc.Launcher$AppClassLoader@5063017f
    -- sun.misc.Launcher$ExtClassLoader@b1526f5d

So, I assume it will work for me only when I pass specific classloader (platform-classloader@e4dc9917) which loads my plugin.
Do you think, is it possible?

6

Hi, there are some problems around backwards compatibility when going with that approach. See also Loading plugins from System CL and parent hierarchy by eyalkoren · Pull Request #516 · elastic/apm-agent-java · GitHub.

What you could do is to fork the agent repo and always use released versions (tags) as the base of your own agent. Would that work for you?

I use some very specific framework in enterprise env.

Is that an Open Source framework or an internal one? If it's the former we could think about adding out-of-the-box support for it.

Cheers,
Felix

7

Hello!
I don't see anything critical about backwards compatibility. I accept all the risks, and if I implement external plugin - it is my responsibility to adapt to internal api changes. It's okay.

Yes, I could fork the agent repo, but I don't think it is the best solution... Only if you have a solid understanding that you will not implement such a feature, I have no options, except make a fork )

The framework I use, better say platform, is proprietary and close source. So I don't think, we could discuss OOTB support.

8

What I did is allowed loading the plugin from the system CL or its parent.
So this means two options:

  1. Try locating the plugin jar in your <jre-home>/lib/ext dir. The sun.misc.Launcher$ExtClassLoader should pick it up.
  2. Try adding it to the application classpath - probably through one of the many WebSphere config XMLs or through the admin console
9

Thank you for explanation! Yes, it sounds good.

Unfortunately, I have some restrictions. I can't just put some jar on disk on all the nodes of production env, also I would like to avoid any websphere config changes..
I can only use platform classloader, which looks for classes in database, where I can put my classes.

So, yes, my case is edge case, super specific.
I'll make a fork.
Thank you guys again! Awesome work, great agent!

1 Like
10

I also think that this is the way to go in this case. BUT: To make your life easier, I strongly encourage you to only add stuff in the https://github.com/elastic/apm-agent-java/tree/master/apm-agent-plugins folder and not to modify any other code. If you feel like there is some core functionality missing, I suggest making a PR upstream. This will ensure smooth updates to newer versions of the agent. However, you will still need to adapt your plugin to changes in the core. But the good thing is that you will get compile errors as opposed to runtime errors.

11

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.