Java APM with Tomcat & Quartz

Elastic Observability APM
1

Hello Elastic Team,

I would test APM java with Tomcat application. My code use Quartz Job scheduler with JDBC connection (MariadDB).
I've configured elk java agent, adding export

CATALINA_OPTS="$CATALINA_OPTS -javaagent:/path/to/elastic-apm-agent-.jar"

to $CATALINA_BASE/bin/setenv.sh.

At startup, I've an error :

2018-07-11 11:55:38,839 [QuartzScheduler_ekContractScheduler-v2-dev1531302937988_MisfireHandler] [ERROR] [o.q.impl.jdbcjobstore.JobStoreTX] - MisfireHandler: Error handling misfires: Failed to obtain DB connection from data source 'ekds': java.lang.NoClassDefFoundError: co/elastic/apm/jdbc/StatementInstrumentation
org.quartz.JobPersistenceException: Failed to obtain DB connection from data source 'ekds': java.lang.NoClassDefFoundError: co/elastic/apm/jdbc/StatementInstrumentation
at org.quartz.impl.jdbcjobstore.JobStoreSupport.getConnection(JobStoreSupport.java:787)
at org.quartz.impl.jdbcjobstore.JobStoreTX.getNonManagedTXConnection(JobStoreTX.java:71)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.doRecoverMisfires(JobStoreSupport.java:3231)
at org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.manage(JobStoreSupport.java:3997)
at org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.run(JobStoreSupport.java:4018)
Caused by: java.lang.NoClassDefFoundError: co/elastic/apm/jdbc/StatementInstrumentation

I've try to add apm-jdbc-plugin-0.6.0.jar into $CATALINA_BASE/lib, no success.

Why co/elastic/apm/jdbc/StatementInstrumentation isn't in the classpath ?

Thanks,

Christophe.

2

Hi Christophe,
Thanks for trying out the agent!

A few questions:

  • Is your application using an OSGi container or something similar?
  • How do you deploy the quartz implementation? Is it inside your war file under WEB-INF/lib?
  • Which quarz version are you using?
  • Which connection pool are you using?

Felix

3

Hi Felix,

First of all : Great project :slight_smile:

  • Application does not use OSGI. We use :
    Jersey (with osgi-resource-locator in dependence)
    jersey-hk2
    spring
  • Quartz is deployed into the WAR (WEB-INF/lib/)
  • Quartz version is v2.3.0
  • Connection pool is HikariCP v3.1.0

Regards,

Christophe.

4

What do you need the osgi-resource-locator for? Are you referring to http://mvnrepository.com/artifact/org.glassfish.jersey.inject/jersey-hk2/2.27?

Could you send me a small application which reproduces the problem? It would be really awesome if you could do this as I can't reproduce that problem on Tomcat.

5

This is my dependencies :

<dependency>
            <groupId>org.glassfish.jersey.ext</groupId>
            <artifactId>jersey-bean-validation</artifactId>
        </dependency>
        <dependency>
            <groupId>org.glassfish.jersey.containers</groupId>
            <artifactId>jersey-container-servlet</artifactId>
        </dependency>
        <dependency>
            <groupId>org.glassfish.jersey.media</groupId>
            <artifactId>jersey-media-json-jackson</artifactId>
        </dependency>
        <dependency>
            <groupId>org.glassfish.jersey.media</groupId>
            <artifactId>jersey-media-multipart</artifactId>
        </dependency>
        <dependency>
            <groupId>org.glassfish.jersey.inject</groupId>
            <artifactId>jersey-hk2</artifactId>
        </dependency>
        <dependency>
            <groupId>org.glassfish.jersey.ext</groupId>
            <artifactId>jersey-spring4</artifactId>
        </dependency>

=> Jersey version is 2.27

org.glassfish.hk2:osgi-resource-locator:jar:1.0.1 come from :

[INFO] +- org.glassfish.jersey.ext:jersey-bean-validation:jar:2.27:compile
[INFO] |  |  |  +- org.glassfish.hk2.external:javax.inject:jar:2.5.0-b42:compile
[INFO] |  |  |  +- org.glassfish.jersey.core:jersey-common:jar:2.27:compile
[INFO] |  |  |  |  +- javax.annotation:javax.annotation-api:jar:1.2:compile
[INFO] |  |  |  |  \- org.glassfish.hk2:osgi-resource-locator:jar:1.0.1:compile

I'll forward your light application request to my dev team but I don't have many hopes..

Chris.

6

Just as a clarification - does the problem occur on Tomcat or on Glassfish? Glassfish is currently not supported and shows the exact same symptoms (NoClassDefFoundError). This is because the OSGi container does not allow access to the agent classes.

What the agent currently does to support other OSGi environments, such as WebSpere, is to set the system property org.osgi.framework.bootdelegation=co.elastic.apm.*. But that somehow does not work on Glassfish and I have yet to find the reason.

7

Hello,

Problem occur on Tomcat, Glassfish is only the GroupID for jersey because is a Glassfish project (https://jersey.github.io). :wink:

However, may be this is a requirement (agentClass access) with osgi-resource-locator even with Tomcat ?

8

Hey, I've recently been adding support for Glassfish. If we are lucky it fixes your problem as well. Could you try with this snapshot version of the agent:

elastic-apm-agent-0.7.0-SNAPSHOT.jar.zip

9

Hi Felix,

Yes it's working with this new version :slight_smile:
Thanks a lot.

I'll continue the agent integration test.

Chris.

10

Hey, that's great!

I can't promise any release date however, since it requires a change in a library I'm using (Byte Buddy).

Thanks for the feedback,
Felix

11

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.