My APM server is up and running. I can see on kibana that i am able to connect to apm server. But when i tried to run the tomcat i found that my application which is deployed on tomcat is not running, i cannot access my application, only tomcat process is up.
So i went through the tomcat catalina.out file and i found these logging in it :
2018-11-20 12:42:27.965 [apm-reporter] INFO co.elastic.apm.report.IntakeV2ReportingEventHandler - Backing off for 36 seconds (±10%)
INFO: Illegal access: this web application instance has been stopped already. Could not load com.google.common.base.MoreObjects$ToStringHelper. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1564)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1523)
at com.google.common.base.MoreObjects.toStringHelper(MoreObjects.java:95)
at com.google.common.cache.CacheStats.toString(CacheStats.java:268)
at com.company.util.cache.provider.GuavaCacheProvider$2.run(GuavaCacheProvider.java:177)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
Note : If i remove the APM configuration from setenv.sh file my application is accessible and i could not
find any of the above error.
Now their is another finding. I thought that their might be some issue with my tomcat version which was 7.x. So, i tried with another product which uses tomcat 8.5, even then the APM monitoring does work.
Please help me where am i doing the mistake.
This time i get following error:
21-Nov-2018 08:26:46.336 SEVERE [localhost-startStop-1] org.apache.catalina.startup.ContextConfig.processAnnotationsJar Unable to process Jar entry [META-INF/versions/9/module-info.class] from Jar [file:/home/webtech/elastic-apm-agent-1.0.1.jar] for annotations
org.apache.tomcat.util.bcel.classfile.ClassFormatException: Invalid byte tag in constant pool: 19
at org.apache.tomcat.util.bcel.classfile.Constant.readConstant(Constant.java:97)
at org.apache.tomcat.util.bcel.classfile.ConstantPool.<init>(ConstantPool.java:54)
at org.apache.tomcat.util.bcel.classfile.ClassParser.readConstantPool(ClassParser.java:174)
at org.apache.tomcat.util.bcel.classfile.ClassParser.parse(ClassParser.java:83)
at org.apache.catalina.startup.ContextConfig.processAnnotationsStream(ContextConfig.java:2053)
at org.apache.catalina.startup.ContextConfig.processAnnotationsJar(ContextConfig.java:2000)
at org.apache.catalina.startup.ContextConfig.processAnnotationsUrl(ContextConfig.java:1970)
at org.apache.catalina.startup.ContextConfig.processAnnotations(ContextConfig.java:1923)
at org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1163)
at org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:775)
at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:299)
at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:94)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5087)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:752)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:728)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1107)
at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1841)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
The second error doesn't seem to be specific to the agent, but a problem with the Tomcat's embedded BCEL. It may be caused by introducing agent classes though containing bytecode that BCEL fails to parse. The solution I see when searching for that is upgrading your Tomcat to 8.5.12 or higher
As for the first error- 2018-11-20 12:42:27.965 [apm-reporter] INFO co.elastic.apm.report.IntakeV2ReportingEventHandler - Backing off for 36 seconds (±10%) indicates that the agent cannot connect to the APM server, so it backs off. This, however can't explain the problem you see. What are the Tomcat and Java versions in this case?
For the second issue i referred this link. So i thought the tomcat 8.5.8 will work. Any possibility that i can do it without upgrading my tomcat.
And regarding the agent not being able to connect to APM server, my APM server shows me the status that it is running and i can easily telnet from my agent server to APM server and currently i am successfully pushing the tomcat log to elastic(APM Server).
You are right- you are absolutely within the supported matrix we stated, the Tomcat 8.5.8 thing is something we need to further check. However, even if we reproduce it, we are likely to decide not to dig in to see what exact annotation is causing that and change our code just to support specific version with a known issue. But we should update the documented supported matrix at least.
I am not sure whether the problem with the tomcat 7 was the same one (although many cases I found online for the BCEL problem were about the combination of tomcat 7 with jdk 8). If you want to go back to this one, try to look further into the logs, as I couldn't find anything I can rely on.
And lastly- what I said about 2018-11-20 12:42:27.965 [apm-reporter] INFO co.elastic.apm.report.IntakeV2ReportingEventHandler - Backing off for 36 seconds (±10%) is that there was a temporary problem with agent communicating the server, maybe even because you started the server after the agent. Which APM server version are you using BTW?
Here are some other logs i think you can find useful :
Exception in thread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0" java.lang.VerifyError: (class: com/mchange/v2/c3p0/impl/NewProxyStatement, method: execute signature: (Ljava/lang/String;)Z) Illegal type in constant pool
at com.mchange.v2.c3p0.impl.NewProxyConnection.createStatement(NewProxyConnection.java:97)
at com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnection(DefaultConnectionTester.java:138)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1.testPooledConnection(C3P0PooledConnectionPool.java:184)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1.refurbishResourceOnCheckin(C3P0PooledConnectionPool.java:138)
at com.mchange.v2.resourcepool.BasicResourcePool.attemptRefurbishResourceOnCheckin(BasicResourcePool.java:1132)
at com.mchange.v2.resourcepool.BasicResourcePool.access$100(BasicResourcePool.java:32)
at com.mchange.v2.resourcepool.BasicResourcePool$6.run(BasicResourcePool.java:862)
at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:368)
SEVERE: The web application [/dummy] appears to have started a thread named [AsyncAppender-Dispatcher-Thread-9] but has failed to stop it. This is very likely to create a memory leak.
This may be useful indeed. Looks like our instrumentation of this c3p0 Statement results in invalid bytecode.
Can you tell the exact version of the c3p0 library you are using and which Java version it was compiled with?
From what I see it seems very old, maybe even compiled with Java 1.4?
We should release a bugfix soon that addresses libs compiled with Java older than 5 (not instrumenting them...).
If indeed this is the problem, and the c3p0 wraps a supported JDBC driver - this may work out, so you may want to try it out.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
