# Kibana APM not showing @CaptureSpan and @CaptureTransaction

**URL:** <https://discuss.elastic.co/t/kibana-apm-not-showing-capturespan-and-capturetransaction/173026>\
**Category:** APM\
**Created:** [March 19, 2019, 5:02pm UTC](https://discuss.elastic.co/t/kibana-apm-not-showing-capturespan-and-capturetransaction/173026 "2019-03-19T17:02:39Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![ivanqwam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ivanqwam/32/28102_2.png) [@ivanqwam](https://discuss.elastic.co/u/ivanqwam)\
**Post date:** [March 19, 2019, 5:02pm UTC](https://discuss.elastic.co/t/kibana-apm-not-showing-capturespan-and-capturetransaction/173026/1 "2019-03-19T17:02:39Z")

</div>

**Kibana version 6.5.3** :

**Elasticsearch version 6.5.3** :

**APM Server version 1.4.0** :

**APM Agent language and version 1.4.0 for Java & Tomcat** :

We installed an APM server and Agent for our java app on Tomcat.

We added this to our tomcat/bin.catalina.sh on Debian Linux

```
JAVA_OPTS="-DTOMCAT5 -verbosegc -Xms2G -Xmx8G -noverify -DQES_DIR=$SOFTWARE/conf -Dfile.encoding=UTF-8 -Dlog4j.configuration=file://$SOFTWARE/conf/log4j.properties -Dorg.xml.sax.driver=org.apache.xerces.parsers.SAXParser -Dorg.owasp.esapi.resources=$SOFTWARE/conf -Dlog4j.configurationFile=file://$SOFTWARE/conf/log4j.properties -javaagent:/usr/software/apm-client/elastic-apm-agent-1.4.0.jar -Delastic.apm.service_name=qeswebmonitor -Delastic.apm.server_url=http://localhost:8200 -Delastic.apm.application_packages=com.qwam "

```

We added the needed maven jars to our app.  
We added some annotations such as this one on a couple of methods

```
package com.qwam.qesrest.facet.services

import co.elastic.apm.api.CaptureSpan
import com.qwam.qes.QESChamp
import com.qwam.qes.QESListe_valeur
import com.qwam.qes.QESObject
import com.qwam.qes.QESSession
import com.qwam.qes.QESUtilisateur
import com.qwam.qesrest.core.domain.Constants
import com.qwam.qesrest.facet.domain.Facet
import com.qwam.qesrest.facet.domain.FacetValue

import java.sql.Connection

public class FacetService extends QESObject {

FacetService() {
}

@CaptureSpan("xmlToFacets")
public List xmlToFacets(QESSession ses, Connection connection, QESUtilisateur utilisateur, Hashtable<String,String> facetTranslations, String xml) {

    List facets = []

    println('DEBUG:FacetService:xmlToFacets():Debut')

....
}

```

We can see our requests along with the SQL statement executions in the APM kibana dashboard but

- No trace for @CaptureSpan and @CaptureTransaction, despite we check that the TRACE is in deed in the logs (which says that the method was executed)
- Could not find it in discover (as far as we could see)

The option

```
-Delastic.apm.application_packages=com.qwam 

```

was mentioned in the doc for @CaptureSpan and @CaptureTransaction.  
Should we declare every package such as

```
-Delastic.apm.application_packages=com.qwam.subpackage1,com.qwam.subpackage2

```

or

```
-Delastic.apm.application_packages=com.qwam 

```

is enough ?

Looks like an issue from a Noob, but could not find a solution on my own.

---

<div class="post-metadata">

**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\
**Post date:** [March 20, 2019, 8:00am UTC](https://discuss.elastic.co/t/kibana-apm-not-showing-capturespan-and-capturetransaction/173026/2 "2019-03-20T08:00:00Z")

</div>

Hi and thanks for trying out our APM 🙂

Your root package setting seems OK, no need for sub-packages.

Note that if you capture a Span while there is no active transaction, this span will not be sent and stored. So you should verify that for each request, the first event is traced through `CaptureTransaction` and all subsequent events are captured through `CaptureSpan`. From the code you added, I can't see if this is indeed related to the problem or not.  
For more info see [our data model documentation](https://www.elastic.co/guide/en/apm/get-started/current/apm-data-model.html).

I hope this helps. If not, please set [log level](https://www.elastic.co/guide/en/apm/agent/java/current/config-logging.html#config-log-level) to `TRACE` and upload the log.

Thanks,  
Eyal.

---

<div class="post-metadata">

**Author:** ![ivanqwam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ivanqwam/32/28102_2.png) [@ivanqwam](https://discuss.elastic.co/u/ivanqwam)\
**Post date:** [March 20, 2019, 10:05am UTC](https://discuss.elastic.co/t/kibana-apm-not-showing-capturespan-and-capturetransaction/173026/3 "2019-03-20T10:05:57Z")

</div>

Hi Eyal,

I added @CaptureTransaction atop my main service: Nothing much appears

Added -Delastic.apm.log\_level=TRACE to my JAVA\_OPTS

[Here are the logs](http://demo.qwamci.com/qesdemo/pdftemp/elastic_apm.log)

As you can see there are only Spans for JDBC in there.

---

<div class="post-metadata">

**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\
**Post date:** [March 20, 2019, 11:01am UTC](https://discuss.elastic.co/t/kibana-apm-not-showing-capturespan-and-capturetransaction/173026/4 "2019-03-20T11:01:55Z")

</div>

Thanks for the logs.

I _do_ see `FacetService#xmlToFacets` occurs within a traced transaction (`QESResultat_jsp#doGet`) and not being traced. Your settings look fine, so I can't see what's wrong at the moment. Please upload the startup area of this log, where we log the active instrumentations and matchings, maybe we can find some hints there.

Thanks,  
Eyal.

---

<div class="post-metadata">

**Author:** ![ivanqwam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ivanqwam/32/28102_2.png) [@ivanqwam](https://discuss.elastic.co/u/ivanqwam)\
**Post date:** [March 20, 2019, 11:36am UTC](https://discuss.elastic.co/t/kibana-apm-not-showing-capturespan-and-capturetransaction/173026/5 "2019-03-20T11:36:32Z")

</div>

Here are the

[Full logs](http://demo.qwamci.com/qesdemo/pdftemp/elastic_apm_full.log)

Sorry for lengthy logs since I am not alone on this Tomcat instance.

---

<div class="post-metadata">

**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\
**Post date:** [March 20, 2019, 12:53pm UTC](https://discuss.elastic.co/t/kibana-apm-not-showing-capturespan-and-capturetransaction/173026/6 "2019-03-20T12:53:03Z")

</div>

Looks like the agent doesn't find any type and method match to the `CaptureSpanInstrumentation` although it sees the `application_packages` properly set to `com.qwam`.  
Please add the following code to the `FacetService` constructor, so we can see which class loader loads it:

```auto
        ClassLoader classLoader = getClass().getClassLoader();
        while (classLoader != null) {
            System.out.println(classLoader.getClass().getName());
            classLoader = classLoader.getParent();
        }

```

---

<div class="post-metadata">

**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\
**Post date:** [March 20, 2019, 1:07pm UTC](https://discuss.elastic.co/t/kibana-apm-not-showing-capturespan-and-capturetransaction/173026/7 "2019-03-20T13:07:13Z")

</div>

@ivanqwam and if you are already adding, please also add to the same constructor:

```auto
System.out.println("CaptureSpan class loader: " + CaptureSpan.class.getClassLoader());

```

---

<div class="post-metadata">

**Author:** ![ivanqwam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ivanqwam/32/28102_2.png) [@ivanqwam](https://discuss.elastic.co/u/ivanqwam)\
**Post date:** [March 20, 2019, 1:55pm UTC](https://discuss.elastic.co/t/kibana-apm-not-showing-capturespan-and-capturetransaction/173026/8 "2019-03-20T13:55:41Z")

</div>

I modified my code as requested.

Here is my code

```
package com.qwam.qesrest.facet.services

import co.elastic.apm.api.CaptureSpan
import com.qwam.qes.QESChamp
import com.qwam.qes.QESListe_valeur
import com.qwam.qes.QESObject
import com.qwam.qes.QESSession
import com.qwam.qes.QESUtilisateur
import com.qwam.qesrest.core.domain.Constants
import com.qwam.qesrest.facet.domain.Facet
import com.qwam.qesrest.facet.domain.FacetValue

import java.sql.Connection

public class FacetService extends QESObject {

    FacetService() {
    }

    @CaptureSpan
    public List xmlToFacets(QESSession ses, Connection connection, QESUtilisateur utilisateur, Hashtable<String,String> facetTranslations, String xml) {

        List facets = []

        println('DEBUG:FacetService:xmlToFacets():Debut')
        
        // debug Eyal Elastic
        System.out.println('DEBUG:Eyal:CaptureSpan class loader: ' + CaptureSpan.class.getClassLoader());
        ClassLoader classLoader = getClass().getClassLoader();
        while (classLoader != null) {
            System.out.println('DEBUG:Eyal:class loaded by:' + classLoader.getClass().getName());
            classLoader = classLoader.getParent();
        }

```

which gives this traces

```
2019/03/20 14:33:53 998-qeswebmonitor-demoqwam-TRACE-ResultatService-buildFacetJstl() : Debut|
DEBUG:FacetService:xmlToFacets():Debut
DEBUG:Eyal:CaptureSpan class loader: ParallelWebappClassLoader^M
  context: qeswebmonitor^M
  delegate: false^M
----------> Parent Classloader:^M
java.net.URLClassLoader@617c74e5^M

    DEBUG:Eyal:class loaded by:org.apache.catalina.loader.ParallelWebappClassLoader
    DEBUG:Eyal:class loaded by:java.net.URLClassLoader
    DEBUG:Eyal:class loaded by:sun.misc.Launcher$AppClassLoader
    DEBUG:Eyal:class loaded by:sun.misc.Launcher$ExtClassLoader
    DEBUG:FacetService:xmlToFacets():Fin

```

Trying to guess where you were going I added the apm-agent-api-1.4.0.jar to Tomcat lib (it is currently in myapp/WEB-INF/lib) thinking that Tomcat should now the jar before the app, since APM is outside the app. But this does not change anything

---

<div class="post-metadata">

**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\
**Post date:** [March 20, 2019, 3:35pm UTC](https://discuss.elastic.co/t/kibana-apm-not-showing-capturespan-and-capturetransaction/173026/9 "2019-03-20T15:35:53Z")

</div>

The `WEB-INF/lib` should work just fine. Anyway, it seems it loaded the class from there even after you added to tomcat lib (which makes sense because the `ParallelWebappClassLoader` has the `delegate: false` setting, so it searches locally first). Did you add any other jar to `WEB-INF/lib`?

---

<div class="post-metadata">

**Author:** ![ivanqwam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ivanqwam/32/28102_2.png) [@ivanqwam](https://discuss.elastic.co/u/ivanqwam)\
**Post date:** [March 20, 2019, 3:59pm UTC](https://discuss.elastic.co/t/kibana-apm-not-showing-capturespan-and-capturetransaction/173026/10 "2019-03-20T15:59:31Z")

</div>

I added only apm-agent-api-1.4.0.jar in **WEB-INF/lib**.  
I had an issue with _netty.jar_\* that I had to add in **$CATALINA\_HOME/lib** to avoid a strange **NoClassDefFound** despite they were in **WEB-INF/lib**.  
That is all.

---

<div class="post-metadata">

**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\
**Post date:** [March 21, 2019, 5:11am UTC](https://discuss.elastic.co/t/kibana-apm-not-showing-capturespan-and-capturetransaction/173026/11 "2019-03-21T05:11:17Z")

</div>

> I had an issue with _netty .jar_ \* that I had to add in **$CATALINA\_HOME/lib** to avoid a strange **NoClassDefFound** despite they were in **WEB-INF/lib**.

This may be important - you mean you had this problem when you installed the agent? If so, can you reproduce and send this log?

---

<div class="post-metadata">

**Author:** ![ivanqwam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ivanqwam/32/28102_2.png) [@ivanqwam](https://discuss.elastic.co/u/ivanqwam)\
**Post date:** [March 21, 2019, 8:47am UTC](https://discuss.elastic.co/t/kibana-apm-not-showing-capturespan-and-capturetransaction/173026/12 "2019-03-21T08:47:10Z")

</div>

Good morning Eyal,

I could not reproduce the **NoClassDefFound** about **netty jars** that I had in the install.  
But when I remove the netty jars from **$CATALINA\_HOME/lib** , only relying on those which are in the **WEB-INF/lib** , I get **JSP page error from Tomcat** and nothing works anymore.  
Here is the netty jar list:

```
netty-buffer-4.1.16.Final.jar
netty-codec-4.1.16.Final.jar
netty-codec-http-4.1.16.Final.jar
netty-common-4.1.16.Final.jar
netty-handler-4.1.16.Final.jar
netty-resolver-4.1.16.Final.jar
netty-transport-4.1.16.Final.jar
transport-netty4-client-6.2.1.jar

```

This dependency upon netty is strange because the Maven dependency for APM does not say a word about it.

There is not much explanations in the [logs](http://demo.qwamci.com/qesdemo/pdftemp/elastic_apm_error404.log) when the JSP page error from Tomcat occurs.

---

<div class="post-metadata">

**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\
**Post date:** [March 21, 2019, 9:02am UTC](https://discuss.elastic.co/t/kibana-apm-not-showing-capturespan-and-capturetransaction/173026/13 "2019-03-21T09:02:25Z")

</div>

Well, I can't say at the moment if this is related or not to the tracing failure you originally reported, but this is really bad. Normally you should just be able to add the API jar to your application and the java agent jar to your JVM startup and nothing else.

This log indeed says nothing related. If possible, please remove the `elastic.apm.log_level` setting (to restore to `INFO` level) and restore everything to the original state- your application and Tomcat original setup, Elastic API jar in the app lib and the `javaagent` in the JVM setup. Then clean the logs, start the app, make a few requests and upload **the full log**.

Thanks,  
Eyal.

---

<div class="post-metadata">

**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\
**Post date:** [March 21, 2019, 9:11am UTC](https://discuss.elastic.co/t/kibana-apm-not-showing-capturespan-and-capturetransaction/173026/14 "2019-03-21T09:11:54Z")

</div>

@ivanqwam please paste here the maven dependency you added in order to use our API

---

<div class="post-metadata">

**Author:** ![ivanqwam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ivanqwam/32/28102_2.png) [@ivanqwam](https://discuss.elastic.co/u/ivanqwam)\
**Post date:** [March 21, 2019, 10:42am UTC](https://discuss.elastic.co/t/kibana-apm-not-showing-capturespan-and-capturetransaction/173026/15 "2019-03-21T10:42:05Z")

</div>

Eyal, please forget my last.  
The Error 404 came from a developper who performed a buggy commit.  
You were so fast to respond, that I did not have time to cancel my post.  
We apologize for this Eyal.

I came back to the situation where the netty jar are removed from $CATALINA\_HOME/lib with a fresh restart. I could not reproduce the NoCLassDefFound issue I had on netty during the install.

Here are the [full logs](http://demo.qwamci.com/qesdemo/pdftemp/elastic_apm_without_netty.log)

My maven dependencies came from [here](https://www.elastic.co/guide/en/apm/agent/java/current/public-api.html)

```
<dependency>
<groupId>co.elastic.apm</groupId>
<artifactId>apm-agent-api</artifactId>
<version>1.4.0</version>
</dependency>

```

which leads to this list

```
apm-agent-api-1.4.0.jar 
backport-util-concurrent-3.1.jar 
classworlds-1.1-alpha-2.jar 
classworlds-1.1.jar 
commons-cli-1.0.jar 
commons-codec-1.6.jar 
commons-lang-2.1.jar 
commons-lang3-3.1.jar 
commons-logging-api-1.1.jar 
doxia-sink-api-1.0-alpha-7.jar 
google-collections-1.0.jar 
jsr305-2.0.1.jar 
junit-3.8.1.jar 
junit-3.8.2.jar 
log4j-1.2.12.jar 
surefire-api-2.17.jar 
surefire-booter-2.17.jar 
xbean-reflect-3.4.jar 

```

I will have to switch to something else in the next few days, so I won't be available.  
I will be back in a week or so.

Do not worry if you did not solve my issue, I know it will work sooner or later.  
APM is a great tool and we have already seen so many things and so many hints to improve performances of our app, even without the annotation working that, it showed us how wonderful and easy to set up it can be.

Thank you Eyal for your tenacious help.

---

<div class="post-metadata">

**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\
**Post date:** [March 21, 2019, 11:42am UTC](https://discuss.elastic.co/t/kibana-apm-not-showing-capturespan-and-capturetransaction/173026/16 "2019-03-21T11:42:30Z")

</div>

Thanks for the incredible feedback! It is great to know that you found it valuable!  
Once you're back and available for that, we can start fresh and take it from there.  
Cheers,  
Eyal.

---

<div class="post-metadata">

**Author:** ![ivanqwam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ivanqwam/32/28102_2.png) [@ivanqwam](https://discuss.elastic.co/u/ivanqwam)\
**Post date:** [March 27, 2019, 12:37pm UTC](https://discuss.elastic.co/t/kibana-apm-not-showing-capturespan-and-capturetransaction/173026/17 "2019-03-27T12:37:14Z")

</div>

Hi Eyal,

I am back.

Trying to narrow down the issue, I installed everything on my desktop.  
I have a Tomcat 9.0.5 there and for sure I am alone.

But I cannot get my application deployed on Tomcat.  
Instead, I get and in the [logs](http://demo.qwamci.com/qesdemo/pdftemp/catalina_9.txt)

`Caused by: java.lang.LinkageError: loader (instance of org/apache/catalina/loader/ParallelWebappClassLoader): attempted duplicate class definition for name: "javax/ws/rs/Path"`

---

<div class="post-metadata">

**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\
**Post date:** [March 27, 2019, 1:04pm UTC](https://discuss.elastic.co/t/kibana-apm-not-showing-capturespan-and-capturetransaction/173026/18 "2019-03-27T13:04:43Z")

</div>

Welcome back 🙂

Doesn't seem related to the agent. You may have the JAX-RS jar more than once on your application's classpath, for example - both in the `tomcat/lib` and the `WEB-INF/lib`.

Does it work when the agent is not installed?

---

<div class="post-metadata">

**Author:** ![ivanqwam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ivanqwam/32/28102_2.png) [@ivanqwam](https://discuss.elastic.co/u/ivanqwam)\
**Post date:** [March 27, 2019, 2:18pm UTC](https://discuss.elastic.co/t/kibana-apm-not-showing-capturespan-and-capturetransaction/173026/19 "2019-03-27T14:18:40Z")

</div>

**javax/ws/rs/Path**

was both in

- jersey-core-1.9.jar
- javax.annotation-api-1.2.jar

I removed **javax.annotation-api-1.2.jar**  
I checked $CATALINA\_HOME/lib , no such class there.  
I restarted with the same result.

When restarting Tomcat without

`-javaagent:/usr/software/apm-client/elastic-apm-agent-1.4.0.jar -Delastic.apm.service_name=qeswebmonitor -Delastic.apm.server_url=http://localhost:8200 -Delastic.apm.application_packages=com.qwam`

in the JAVA\_OPTS, the application is deployed normally without this issue.

I agree that it does not seem related to APM, but since it works without it, I lazily turn to you 😳.

---

<div class="post-metadata">

**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\
**Post date:** [March 27, 2019, 4:33pm UTC](https://discuss.elastic.co/t/kibana-apm-not-showing-capturespan-and-capturetransaction/173026/20 "2019-03-27T16:33:35Z")

</div>

Ahh, I see why we can cause that...  
Can you set `jax-rs` in the [instrumentation diablement configuration](https://www.elastic.co/guide/en/apm/agent/java/current/config-core.html#config-disable-instrumentations) and see if the problem disappears? It's just a check, you seem to need this instrumentation, so I will create a fix once it is verified.

Still, I assume there is another jar with this class somewhere (maybe in `jre-home/lib/ext`?), so if you find such and can remove it, it may resolve the problem

[Next page](https://discuss.elastic.co/t/kibana-apm-not-showing-capturespan-and-capturetransaction/173026.md?page=2)