NoClassDefFoundError in Wildlfy 16 for Java APM agent

thiyagarajan-angappa · May 16, 2019, 9:26pm

Kibana version: 6.7.1

Elasticsearch version: 6.7.1

APM Server version: 6.7.1

APM Agent language and version:
Java and 1.6.1

Browser version:
Chrome 74.0.3729.131 (Official Build) (64-bit)

Original install method (e.g. download page, yum, deb, from source, etc.) and version:
deb

Fresh install or upgraded from other version?
Fresh

Is there anything special in your setup?
NO

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
I have developed a simple REST application(with Jersey and Maven) with a few resources. I would like to try using elastic apm and so installed the apm server and agent for java.
Added the following code in resources that returns a text "Hello you!". Just before returning the response, i have used the elastic-apm-attach jar to attach the java agent to the rest resouce call

    @GET
	@Path("/you")
	public String helloyou() {
		Map<String, String> apmConfig = new HashMap<String, String>();
		apmConfig.put("service_name", "apmdemoapp-rest-you");
		apmConfig.put("server_urls", "http://127.0.0.1:8200");
		apmConfig.put("application_packages", "com.rest.apm");
		System.out.println("---> attaching apm");
		ElasticApmAttacher.attach(apmConfig);
		System.out.println("---> attached apm");

		return "Hello you!";
	}

Project Structure

pom.xml

Tomcat 9
The application works fine and i am able to see a transaction created in Kibana's APM section

Wildfly 16
The same application without any code or configuration change fails while calling ElasticApmAttacher.attach(apmConfig); with the following error

STACKTRACE
https://pastebin.com/icc36PGN (too large to paste here)

I could see a few NoClassDefFoundErrors and also an UnsatisfiedLinkError at the end for libattach.so.

tried adding elastic-apm-agent-1.6.1.jar to the classPath in an effort to fix the NoClassDefFoundErrors, but it did not resolve the issue

Can someone help me in understanding this error and why the same project and configuration works without any issues in Tomcat 9 but fails in Wildfly 16

Note: I am totally new to ELK stack and JBoss server and still trying to understand how APM works. Please let me know if more or any specific information is required related to the issue and i am happy to provide in order to get this fixed

Eyal_Koren · May 20, 2019, 7:17am

Hi and welcome to the forum

This is definitely not the right usage for the Attacher jar. The Java agent needs to be installed through any one of several options, one of which is ElasticApmAttacher.attach(). If you choose this option, it needs to be done once at the beginning of your application's code only once. You can also use an alternative installation method as well.

Since you are using a supported technology, your code will be automatically traced.

I hope this helps,
Eyal.

thiyagarajan-angappa · May 20, 2019, 10:31pm

Thank you for the help @Eyal_Koren !

Made changes to the code to call the attacher at the beginning of the application. On start of the server could see the following logs being printed

17:21:14,097 INFO [stdout] (apm-server-healthcheck) 2019-05-20 17:21:14.096 [apm-server-healthcheck] INFO co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is available: {"ok":{"build_date":"2019-04-02T14:55:26Z","build_sha":"5665711214eb3fcb956e7d1c61795e427bf97097","version":"6.7.1"}}
17:21:14,143 INFO [stdout] (Attach Listener) 2019-05-20 17:21:14.143 [Attach Listener] INFO co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM 1.6.1 as Main on Java 1.8.0_201 (Oracle Corporation) Linux 4.18.0-20-generic

After the above logs, there are bunch of NoClassDefFoundError and ClassNotFoundExceptions as shown below

> 17:21:17,381 ERROR [stderr] (ServerService Thread Pool -- 76) java.lang.NoClassDefFoundError: co/elastic/apm/agent/bci/ElasticApmInstrumentation
> 17:21:17,382 ERROR [stderr] (ServerService Thread Pool -- 76) 	at javax.servlet.GenericServlet.init(GenericServlet.java:243)
> 17:21:17,382 ERROR [stderr] (ServerService Thread Pool -- 76) 	at io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:117)
> 17:21:17,382 ERROR [stderr] (ServerService Thread Pool -- 76) 	at org.wildfly.extension.undertow.security.RunAsLifecycleInterceptor.init(RunAsLifecycleInterceptor.java:78)
> 17:21:17,384 ERROR [stderr] (ServerService Thread Pool -- 76) 	at io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:103)
> 17:21:17,384 ERROR [stderr] (ServerService Thread Pool -- 76) 	at io.undertow.servlet.core.ManagedServlet$DefaultInstanceStrategy.start(ManagedServlet.java:303)
> 17:21:17,384 ERROR [stderr] (ServerService Thread Pool -- 76) 	at io.undertow.servlet.core.ManagedServlet.createServlet(ManagedServlet.java:143)
> 17:21:17,385 ERROR [stderr] (ServerService Thread Pool -- 76) 	at io.undertow.servlet.core.DeploymentManagerImpl$2.call(DeploymentManagerImpl.java:583)
> 17:21:17,385 ERROR [stderr] (ServerService Thread Pool -- 76) 	at io.undertow.servlet.core.DeploymentManagerImpl$2.call(DeploymentManagerImpl.java:554)
> 17:21:17,386 ERROR [stderr] (ServerService Thread Pool -- 76) 	at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:42)
> 17:21:17,386 ERROR [stderr] (ServerService Thread Pool -- 76) 	at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
> 17:21:17,386 ERROR [stderr] (ServerService Thread Pool -- 76) 	at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
> 17:21:17,387 ERROR [stderr] (ServerService Thread Pool -- 76) 	at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
> 17:21:17,387 ERROR [stderr] (ServerService Thread Pool -- 76) 	at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
> 17:21:17,387 ERROR [stderr] (ServerService Thread Pool -- 76) 	at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
> 17:21:17,388 ERROR [stderr] (ServerService Thread Pool -- 76) 	at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
> 17:21:17,388 ERROR [stderr] (ServerService Thread Pool -- 76) 	at io.undertow.servlet.core.DeploymentManagerImpl.start(DeploymentManagerImpl.java:596)
> 17:21:17,389 ERROR [stderr] (ServerService Thread Pool -- 76) 	at org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:97)
> 17:21:17,390 ERROR [stderr] (ServerService Thread Pool -- 76) 	at org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:78)
> 17:21:17,391 ERROR [stderr] (ServerService Thread Pool -- 76) 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 17:21:17,391 ERROR [stderr] (ServerService Thread Pool -- 76) 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 17:21:17,392 ERROR [stderr] (ServerService Thread Pool -- 76) 	at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
> 17:21:17,392 ERROR [stderr] (ServerService Thread Pool -- 76) 	at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982)
> 17:21:17,399 ERROR [stderr] (ServerService Thread Pool -- 76) 	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
> 17:21:17,399 ERROR [stderr] (ServerService Thread Pool -- 76) 	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
> 17:21:17,399 ERROR [stderr] (ServerService Thread Pool -- 76) 	at java.lang.Thread.run(Thread.java:748)
> 17:21:17,400 ERROR [stderr] (ServerService Thread Pool -- 76) 	at org.jboss.threads.JBossThread.run(JBossThread.java:485)
> 17:21:17,400 ERROR [stderr] (ServerService Thread Pool -- 76) Caused by: java.lang.ClassNotFoundException: co.elastic.apm.agent.bci.ElasticApmInstrumentation from [Module "javax.servlet.api" version 1.0.0.Final from local module loader @dfd3711 (finder: local module finder @42d3bd8b (roots: /opt/wildfly/wildfly-16.0.0.Final/modules,/opt/wildfly/wildfly-16.0.0.Final/modules/system/layers/base))]
> 17:21:17,401 ERROR [stderr] (ServerService Thread Pool -- 76) 	at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:255)
> 17:21:17,401 ERROR [stderr] (ServerService Thread Pool -- 76) 	at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:410)
> 17:21:17,401 ERROR [stderr] (ServerService Thread Pool -- 76) 	at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:398)
> 17:21:17,402 ERROR [stderr] (ServerService Thread Pool -- 76) 	at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:116)
> 17:21:17,402 ERROR [stderr] (ServerService Thread Pool -- 76)

ElasticApmInstrumentation is a part of elastic-apm-agent-1.6.1.jar if i am not wrong. I have tried adding the jar to

.classpath
MANIFEST.MF
added to WEB-INF/lib

but the error does not seem to go away !

Eyal_Koren · May 21, 2019, 6:43am

Can you try install the agent through the javaagent option?
Just make sure to remove all other agent jars you added to all those places. It shouldn't be in application server classpath, application classpath or any other, just a jar you point to through this command line arg.

Also, please add the code you used for the agent attachment (and any other dependencies on agent code if made such).

Thanks,
Eyal.

thiyagarajan-angappa · May 21, 2019, 9:33pm

I have pushed the project code to GitHub here -> RestAPMDemo

Using -javaagent option
Including the following lines in wildfly's standalone.conf file

.
.
.
# Uncomment to enable the experimental JDK 11 support for ByteBuddy
# ByteBuddy is the default bytecode provider of Hibernate ORM
#JAVA_OPTS="$JAVA_OPTS -Dnet.bytebuddy.experimental=true"

# Uncomment this to run with a security manager enabled
# SECMGR="true"

# Uncomment this in order to be able to run WildFly on FreeBSD
# when you get "epoll_create function not implemented" message in dmesg output
#JAVA_OPTS="$JAVA_OPTS -Djava.nio.channels.spi.SelectorProvider=sun.nio.ch.PollSelectorProvider"

# Uncomment this out to control garbage collection logging
# GC_LOG="true"

# Uncomment and edit to use a custom java.security file to override all the Java security properties
#JAVA_OPTS="$JAVA_OPTS -Djava.security.properties==/path/to/custom/java.security"

export JAVA_OPTS="$JAVA_OPTS -javaagent:/home/user/Downloads/elastic-apm-agent-1.6.1.jar"
export JAVA_OPTS="$JAVA_OPTS -Delastic.apm.service_name=apm-rest-demo-19"
export JAVA_OPTS="$JAVA_OPTS -Delastic.apm.application_packages=com.rest.apm"
export JAVA_OPTS="$JAVA_OPTS -Delastic.apm.server_urls=http://localhost:8200"
export JAVA_OPTS="$JAVA_OPTS -Delastic.apm.log_level=DEBUG"

I could not see any logs related to elastic or apm during server startup. Tried hitting the rest endpoint and recevieved a successful response too but could not see any new services or transactions being logged in APM Services in Kibana

Wildfly startup logs
Server startup logs

Eyal_Koren · May 22, 2019, 6:53am

I can't see why this should matter, but did you also try to make all those JAVA_OPTS assignments without the export?

thiyagarajan-angappa · May 22, 2019, 7:20am

Yes. I did remove the export and started the server but did not see any difference.

also tried configuring the way it is mentioned under domain mode. still got the same results

Eyal_Koren · May 22, 2019, 8:11am

Please try to see whether this configuration is applied at all.
For example, add a test code to your app to see whether the System properties are set- invocation of System.getProperty("elastic.apm.service_name") should return apm-rest-demo-19 and not null.

thiyagarajan-angappa · May 24, 2019, 4:05am

I was trying in eclipse and totally forgot to update the VM arguments in the server's launch configuration. The -javaagent option worked perfectly there after ! Checked in kibana and was able to see the service and transaction being logged ! thank you very much for patiently helping me on that. The configuration changes done in standalone.conf is also working as expected

As a next step, tried using ElasticApmAttacher.attach() to perform apm by following the steps here

Updated Git with lastes code - Repo ( upatetes to pom.xml and application class)

This still resulted in the the ClassNotFoundExceptions -> Error Log

suikast42 · May 25, 2019, 1:19pm

I am using it with wf 16 without any problems.
But had same issues if I set the java opts in standalone.conf.
Intelij overrides the java opts.
I set the java opts in standalone.bat or .sh instead of the conf.

Eyal_Koren · May 26, 2019, 7:40am

In general, the programmatic API is more suitable for standalone Java applications not running on a servlet-container or app-server. We will add a such a note to the documentation page.

If editing the startup script is a valid option, the most straightforward way to install an agent would be using the -javaagent flag. Otherwise, if you can use the the remote attache option.

system · June 16, 2019, 3:40am

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.