Tomcat elastic-apm-agent, and application version in context path fails on ProtectionDomain

Elastic Observability APM
#1

I'm currently configuring a Java app in tomcat7 to use elastic-apm-agent.
Java - elastic-apm-agent-1.6.1
Tomcat is running on a linux ec2-instance, its' currently at tomcat-7.0.94.

I've copied the agent in, added the configuration to my tomcat7.conf, and started the tomcat service, and in catalina.out i get:
2019-06-10 07:38:47.860 [localhost-startStop-1] ERROR
co.elastic.apm.agent.bci.bytebuddy.CustomElementMatchers - Cannot read implementation
version based on ProtectionDomain ProtectionDomain
(file:/var/lib/tomcat7/webapps/portal%23%23k000226/WEB-INF/lib/httpclient-4.5.2.jar )
WebappClassLoader
context: /portal##k000226
delegate: false
repositories:
/WEB-INF/classes/
----------> Parent Classloader:
java.net.URLClassLoader@2f490758

 <no principals>
 java.security.Permissions@31421bfe (
 ("java.io.FilePermission" "/var/lib/tomcat7/webapps/portal##k000226/WEB-INF/lib/httpclient-4.5.2.jar" "read")
)

If i remove the application version from the deployed war, this problem goes away. It seems like the filename is corrupted when checking for ProtectionDomain? Is there any other way I can get this to work and still retain the application version in the context path?

#2

Hi and welcome to the forum :slight_smile:

I think I better change the behavior so that if version is not found, we default to instrument, so we at least attempt (in this case there will be no harm in any case).

Still, can you please provide the full related log entry? The log section you added should be followed by an Exception and stack trace.

#3

Ok yep I can get that tomorrow, time zone beat me today!
The bit that didn’t format above with the same file but %23 instead of hash where it said protectiondomain I think is the root cause, it looks to have URL encoded a file path, that’s not right I think? Certainly if I try to ‘ls’ that path, the shell can’t locate it....

#4

It may very well be the case, so decoding is an optional solution. I just want to see what we actually logged, as it should direct us.

#5 
2019-06-10T07:38:11.439+1000: [CMS-concurrent-abortable-preclean-start]
2019-06-10 07:38:14.108 [localhost-startStop-1] ERROR co.elastic.apm.agent.bci.bytebuddy.CustomElementMatchers - Cannot read implementation version based on ProtectionDomain ProtectionDomain  (file:/var/lib/tomcat7/webapps/v4%23%23k000306/WEB-INF/lib/httpclient-4.5.6.jar <no signer certificates>)
 WebappClassLoader^M
  context: /v4##k000306^M
  delegate: false^M
  repositories:^M
    /WEB-INF/classes/^M
----------> Parent Classloader:^M
java.net.URLClassLoader@2f490758^M

 <no principals>
 java.security.Permissions@2a51013e (
 ("java.io.FilePermission" "/var/lib/tomcat7/webapps/v4##k000306/WEB-INF/lib/httpclient-4.5.6.jar" "read")
)


java.io.FileNotFoundException: /var/lib/tomcat7/webapps/v4%23%23k000306/WEB-INF/lib/httpclient-4.5.6.jar (No such file or directory)
        at java.util.zip.ZipFile.open(Native Method)
        at java.util.zip.ZipFile.<init>(ZipFile.java:225)
        at java.util.zip.ZipFile.<init>(ZipFile.java:155)
        at java.util.jar.JarFile.<init>(JarFile.java:166)
        at java.util.jar.JarFile.<init>(JarFile.java:103)
        at co.elastic.apm.agent.bci.bytebuddy.CustomElementMatchers.readImplementationVersionFromManifest(CustomElementMatchers.java:152)
        at co.elastic.apm.agent.bci.bytebuddy.CustomElementMatchers.access$100(CustomElementMatchers.java:44)
        at co.elastic.apm.agent.bci.bytebuddy.CustomElementMatchers$2.matches(CustomElementMatchers.java:128)
        at co.elastic.apm.agent.bci.bytebuddy.CustomElementMatchers$2.matches(CustomElementMatchers.java:125)
        at co.elastic.apm.agent.bci.ElasticApmAgent$4.matches(ElasticApmAgent.java:187)
        at co.elastic.apm.agent.shaded.bytebuddy.agent.builder.AgentBuilder$Default$Transformation$Simple.matches(AgentBuilder.java:9983)
        at co.elastic.apm.agent.shaded.bytebuddy.agent.builder.AgentBuilder$Default$Transformation$Simple.resolve(AgentBuilder.java:9996)
        at co.elastic.apm.agent.shaded.bytebuddy.agent.builder.AgentBuilder$Default$Transformation$Compound.resolve(AgentBuilder.java:10252)
        at co.elastic.apm.agent.shaded.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.resolve(AgentBuilder.java:10584)
        at co.elastic.apm.agent.shaded.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doTransform(AgentBuilder.java:10551)
        at co.elastic.apm.agent.shaded.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:10514)
        at co.elastic.apm.agent.shaded.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.access$1500(AgentBuilder.java:10280)
        at co.elastic.apm.agent.shaded.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$LegacyVmDispatcher.run(AgentBuilder.java:10889)
        at co.elastic.apm.agent.shaded.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$LegacyVmDispatcher.run(AgentBuilder.java:10836)
        at java.security.AccessController.doPrivileged(Native Method)
        at co.elastic.apm.agent.shaded.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:10437)
        at sun.instrument.TransformerManager.transform(TransformerManager.java:188)
        at sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:428)
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:3222)
        at org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:1409)
        at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1897)
        at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1771)
        at org.apache.http.impl.nio.client.HttpAsyncClients.createSystem(HttpAsyncClients.java:70)
        at org.springframework.http.client.HttpComponentsAsyncClientHttpRequestFactory.<init>(HttpComponentsAsyncClientHttpRequestFactory.java:58)
        at com.pds.common.http.AbstractHttpClient.<clinit>(AbstractHttpClient.java:174)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:147)

different app, same issue... (hence portal from above is now v4)
In case it is important - this is tomcat 7.0.94...

#6

Thanks for log. It is indeed the encoding.
Please test this snapshot and see if it fixes the problem.
Thanks,
Eyal.

#7

I'd love to test it, but the link doesn't seem to be valid?

I tried the snapshot link and got elastic-apm-agent-1.6.2-20190611.093948-46.jar, but that didnt seem to include your fix, or didn't work if it did... Having a look at the code, i'd say it didn't include your fix...

I tried to build it but I'm using JDK8 so its not so simple :confused:

Cheers,

Paul.

#8

Sorry, corrupted link :man_facepalming:
Please try this one instead.
In any case, we will probably release 1.7.0 this week, which should fix this.

#9

Hi Eyal,
Thanks for the new link - i tried that new snapshot and it's resolved the issue!

Cheers,

Paul.

1 Like
#10

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.