# Problem with distributing tracing for Jakarta 10 and payara 6 **URL:** \ **Category:** APM\ **Tags:** java\ **Created:** [January 8, 2025, 7:59am UTC](https://discuss.elastic.co/t/problem-with-distributing-tracing-for-jakarta-10-and-payara-6/372920 "2025-01-08T07:59:41Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![ciment](https://avatars.discourse-cdn.com/v4/letter/c/c37758/32.png) [@ciment](https://discuss.elastic.co/u/ciment)\ **Post date:** [January 8, 2025, 7:59am UTC](https://discuss.elastic.co/t/problem-with-distributing-tracing-for-jakarta-10-and-payara-6/372920/1 "2025-01-08T07:59:41Z")
Hi, I have a problem with distributing tracing after migration of my application (few microservices) from Payara 5.201 to 6.2024.12. During migration we have already updated application to Jakarta 10 (previous Java EE 8), to JDK 21 (previous JDK 11). **Kibana version** : v 8.17.0 **Elasticsearch version** : v 8.17.0 **APM Server version** : v 8.17.0 **APM Agent language and version** : 1.52.1 In Kibana - distributed tracing is broken for transaction. I see only this transaction.name like - _ServletContainer#doPos_t or _ServletContainer#doGet_ After clicking on transaction I cannot see other REST api calls or database span and other spans which were related to this transaction. Elastic APM agent is attached to payara via jvm options and data are sent to apm server directly. Previously my distributing traces in Kibana: ![Screenshot 2025-01-03 at 12.49.12](https://us1.discourse-cdn.com/elastic/original/3X/4/c/4c820286f0b4540ef2163d92a5c5cdcd75fb03d8.png) Now - No spans When I start my Payara server with application, I see in my payara server.log this error: ```auto 2024-12-19 14:31:34,684 [http-thread-pool::http-listener-1(23)] WARN co.elastic.apm.agent.impl.transaction.AbstractSpanImpl - End has already been called: 'ServletContainer#doGet' 00-fce4d5a73d9730ceda1fa9dfa23a6cb6-38d98ac57c88f2bd-01 (55550545) java.lang.Throwable: null at co.elastic.apm.agent.impl.transaction.AbstractSpanImpl.end(AbstractSpanImpl.java:582) ~[elastic-apm-agent-1.52.1.jar:1.52.1] at co.elastic.apm.agent.impl.transaction.AbstractSpanImpl.end(AbstractSpanImpl.java:549) ~[elastic-apm-agent-1.52.1.jar:1.52.1] at co.elastic.apm.agent.servlet.ServletTransactionHelper.onAfter(ServletTransactionHelper.java:248) ~[elastic-apm-agent-1.52.1.jar:1.52.1] at co.elastic.apm.agent.servlet.ServletApiAdvice.onExitServlet(ServletApiAdvice.java:277) ~[elastic-apm-agent-1.52.1.jar:1.52.1] at co.elastic.apm.agent.servlet.JakartaServletApiAdvice.onExitServletService(JakartaServletApiAdvice.java:45) ~[elastic-apm-agent-1.52.1.jar:1.52.1] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:213) ~[web-core.jar:?] at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:257) ~[web-core.jar:?] at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:166) ~[web-core.jar:?] at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:757) ~[web-core.jar:?] at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:577) ~[web-core.jar:?] at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:99) ~[web-glue.jar:?] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:158) ~[web-core.jar:?] at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:372) ~[web-core.jar:?] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:239) ~[web-core.jar:?] at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:520) ~[kernel.jar:?] at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:217) ~[kernel.jar:?] at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:174) ~[nucleus-grizzly-all.jar:?] at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:153) ~[nucleus-grizzly-all.jar:?] at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:196) ~[nucleus-grizzly-all.jar:?] at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:88) ~[nucleus-grizzly-all.jar:?] at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:246) ~[nucleus-grizzly-all.jar:?] at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:178) ~[nucleus-grizzly-all.jar:?] at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:118) ~[nucleus-grizzly-all.jar:?] at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:96) ~[nucleus-grizzly-all.jar:?] at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:51) ~[nucleus-grizzly-all.jar:?] at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:510) ~[nucleus-grizzly-all.jar:?] at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:82) ~[nucleus-grizzly-all.jar:?] at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:83) ~[nucleus-grizzly-all.jar:?] at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:101) ~[nucleus-grizzly-all.jar:?] at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:535) ~[nucleus-grizzly-all.jar:?] at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:515) ~[nucleus-grizzly-all.jar:?] at java.lang.Thread.run(Thread.java:1583) [?:?] 2024-12-19 14:31:40,493 [http-thread-pool::http-listener-1(24)] ``` I found in elastic java agent documentation that Payara 6 is not supported or tested. How to resolve this issue? Is there any option how to fix this issue? Thanks in advance. ---
**Author:** ![Sylvain\_Juge](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylvain_juge/32/55521_2.png) [@Sylvain\_Juge](https://discuss.elastic.co/u/Sylvain_Juge)\ **Post date:** [January 8, 2025, 8:54am UTC](https://discuss.elastic.co/t/problem-with-distributing-tracing-for-jakarta-10-and-payara-6/372920/2 "2025-01-08T08:54:43Z")
Hi, thanks for reporting this ! The error message here indicates that there could be a bug in the agent, in particular it seems that the lifecycle of the transaction (which is here from servlet instrumentation) does not match what we usually expect, hence the message about already ended span/transaction. This is very probably related to your update of Payara 6. - Can you easily reproduce it ? - Is there anything special with this particular transaction or does it happens with all of them ? For example, this may happen only on async servlets, or there is something special in the servlet/filter configuration. - can you provide a simple application that helps reproduce it ? ---
**Author:** ![ciment](https://avatars.discourse-cdn.com/v4/letter/c/c37758/32.png) [@ciment](https://discuss.elastic.co/u/ciment)\ **Post date:** [January 9, 2025, 7:59am UTC](https://discuss.elastic.co/t/problem-with-distributing-tracing-for-jakarta-10-and-payara-6/372920/3 "2025-01-09T07:59:22Z")
> [@ciment](#): > > _ServletContainer#doPos_t or _ServletContainer#doGet_ I have already prepared simple application where you can reproduce this issue. > **[GitHub - ciment7/jakarta-test](https://github.com/ciment7/jakarta-test)** > > Contribute to ciment7/jakarta-test development by creating an account on GitHub. For simplicity - 1 transaction and 1 span (via @Traced annotation) APM with Payara 5: ![Screenshot 2025-01-08 at 15.53.12](https://us1.discourse-cdn.com/elastic/original/3X/8/3/83a124489822cf69209e4c671c9fd2d9b7ffd30d.png) APM with payara 6: ![Screenshot 2025-01-08 at 15.17.00](https://us1.discourse-cdn.com/elastic/original/3X/8/3/8386818bff2fdc65ee590347da0757f2f51f6134.png) I can summary here what is not working on payara 6. 1. transaction.name - only see ServletContane#doGet instead of proper name of resource#method 2. traced annotation is not visible 3. in response header - (see TraceIdHeaderResponseFilter) - for async REST call the header X-Trace-Id is always empty (on payara 5 - trace id is always added to response header) Thanks. ---
**Author:** ![Sylvain\_Juge](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylvain_juge/32/55521_2.png) [@Sylvain\_Juge](https://discuss.elastic.co/u/Sylvain_Juge)\ **Post date:** [January 10, 2025, 4:14pm UTC](https://discuss.elastic.co/t/problem-with-distributing-tracing-for-jakarta-10-and-payara-6/372920/4 "2025-01-10T16:14:54Z")
I have managed to reproduce your setup, I have observed the following regarding transaction name: - the first invocation of `HelloResource#hello` after the domain startup is properly named - the following invocations the same URL are all captured as `ServletContainer#doGet` Restarting the domain allows to reproduce this behavior. In both cases, the method annotated with `@Traced` is not being captured as it should, and I don't know yet why. ---
**Author:** ![Sylvain\_Juge](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylvain_juge/32/55521_2.png) [@Sylvain\_Juge](https://discuss.elastic.co/u/Sylvain_Juge)\ **Post date:** [January 13, 2025, 2:28pm UTC](https://discuss.elastic.co/t/problem-with-distributing-tracing-for-jakarta-10-and-payara-6/372920/5 "2025-01-13T14:28:23Z")
I think that I've found the issue for the transaction name: You are using JAX-RS to create your endpoints, which is most often executed within a servlet container. For the first invocation of the JAX-RS resource, a "servlet transaction" is captured and the JAZ-RS instrumentation overrides the name of the transaction. For the next invocations, there is no such "servlet transaction" created as the invocation of the JAX-RS resource is made directly, however the agent does not currently creates transactions for such "stand-alone JAX-RS" execution as described in [this issue](https://github.com/elastic/apm-agent-java/issues/1752) and as written in the comments fixing that could be tricky. For the method annotated with `@Traced` this is still unexpected, as even in the first method invocation we can't see the method properly traced, whereas it should be. * * * As an alternative approach here I tried doing the same with the opentelemetry-java-instrumentation project without any success, when adding the `-javaagent` with the agent jar it prevents the server from starting and I did not managed to get any hint on the pausible cause. * * * As yet another alternative, it seems there is a partial support for otel tracing directly into Payara which can be enabled with `-Dotel.sdk.disabled=false` in the JVM options, using `otel.exporter.otlp.endpoint` and `otel.exporter.otlp.headers` to provide apm-server URL and authentication did not allowed me to send any data to apm-server however. ---
**Author:** ![ciment](https://avatars.discourse-cdn.com/v4/letter/c/c37758/32.png) [@ciment](https://discuss.elastic.co/u/ciment)\ **Post date:** [January 16, 2025, 11:37am UTC](https://discuss.elastic.co/t/problem-with-distributing-tracing-for-jakarta-10-and-payara-6/372920/6 "2025-01-16T11:37:10Z")
Hi @Sylvain_Juge. I have the same behaviour for elastic apm agent. Name for first transaction is ok, other transactions have broken name. I tried open-telemetry agent. I have no problem to start Payara with this agent, as you mentioned before. Distributing tracing is not working for 100%, I still have a issue with asynchronous requests... but this is another topic. 馃檪 I have a question about elastic java agent and this issue which is in this thread. Should I raise some ticket for Elastic Team or what is your opinion how to resolve this bug? Or Could you share this bug with your team members? Thanks. ---
**Author:** ![Sylvain\_Juge](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylvain_juge/32/55521_2.png) [@Sylvain\_Juge](https://discuss.elastic.co/u/Sylvain_Juge)\ **Post date:** [January 16, 2025, 12:53pm UTC](https://discuss.elastic.co/t/problem-with-distributing-tracing-for-jakarta-10-and-payara-6/372920/7 "2025-01-16T12:53:30Z")
I've opened [Paraya 6 JAX-RS invalid transaction name + no span captured with @Trace 路 Issue #3943 路 elastic/apm-agent-java 路 GitHub](https://github.com/elastic/apm-agent-java/issues/3943) to track those issues and not to forget about it. From what I understand this might be limited to JAX-RS resources deployed in payara, which could explain why it wasn't reported earlier. If you have other jax-rs applications that are using a servlet architecture (and not the one provided by the container), it could be worth to investigate if those are also affected or if it's only for this particular use-case. When using opentelemetry java agent, if there are any issues I would recommend filing an issue in the opentelemetry java instrumentation repository where the instrumentation is being done. Providing ready-to-use examples like you already did here to reproduce issues is key to ensure they are quickly fixed. ---
**Author:** ![Sylvain\_Juge](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylvain_juge/32/55521_2.png) [@Sylvain\_Juge](https://discuss.elastic.co/u/Sylvain_Juge)\ **Post date:** [January 20, 2025, 3:29pm UTC](https://discuss.elastic.co/t/problem-with-distributing-tracing-for-jakarta-10-and-payara-6/372920/8 "2025-01-20T15:29:03Z")
I tried today to do another attempt with OpenTelemetry agent and managed to make it at least partially working: - using the opentelemetry instrumentation agent - using a local collector without authentication that forwards to the backend, otherwise I wasn't able to make the connection work from the instrumentation agent even when bypassing the payara truststore we get some weird protocol errors With that setup, at least you should get a consistent transaction/span name. In order to capture custom spans, we have to use the `@WithSpan` annotation as described here: [Annotations | OpenTelemetry](https://opentelemetry.io/docs/zero-code/java/agent/annotations/), and the required dependency might even be shipped with Payara so using a `provided` scope should work. But, when trying to do that, I get the following exception which seems to be related to the way Payara generates proxies/classes for the annotated classes: ```auto StandardWrapperValve[com.ciment.test.jakartatest.HelloApplication]: Servlet.service() for servlet com.ciment.test.jakartatest.HelloApplication threw exception org.jboss.weld.exceptions.DefinitionException: WELD-001537: An InjectionTarget is created for a class fish.payara.microprofile.telemetry.tracing._WithSpanMethodInterceptorBean_Serializable which does not have any appropriate constructor. at org.jboss.weld.util.InjectionTargets.createNonProducibleInjectionTarget(InjectionTargets.java:76) at org.jboss.weld.util.InjectionTargets.createNonProducibleInjectionTarget(InjectionTargets.java:48) at org.jboss.weld.manager.InjectionTargetFactoryImpl.chooseInjectionTarget(InjectionTargetFactoryImpl.java:126) at org.jboss.weld.manager.InjectionTargetFactoryImpl.createInjectionTarget(InjectionTargetFactoryImpl.java:88) at org.jboss.weld.bean.ManagedBean.(ManagedBean.java:102) at org.jboss.weld.bean.InterceptorImpl.(InterceptorImpl.java:64) at org.jboss.weld.bean.InterceptorImpl.of(InterceptorImpl.java:60) at org.glassfish.weld.services.JCDIServiceImpl.createInterceptorInstance(JCDIServiceImpl.java:468) at com.sun.ejb.containers.BaseContainer.createEjbInterceptors(BaseContainer.java:1800) at com.sun.ejb.containers.BaseContainer.createEmptyContextAndInterceptors(BaseContainer.java:1689) at com.sun.ejb.containers.BaseContainer.createEjbInstanceAndContext(BaseContainer.java:1703) at com.sun.ejb.containers.StatelessSessionContainer.createStatelessEJB(StatelessSessionContainer.java:419) at com.sun.ejb.containers.StatelessSessionContainer$SessionContextFactory.create(StatelessSessionContainer.java:651) at com.sun.ejb.containers.util.pool.NonBlockingPool.getObject(NonBlockingPool.java:219) at com.sun.ejb.containers.StatelessSessionContainer._getContext(StatelessSessionContainer.java:395) at com.sun.ejb.containers.BaseContainer.getContext(BaseContainer.java:2607) ``` The implementation of OpenTelemetry tracing in payara seems to be implemented [here](https://github.com/payara/Payara/tree/0c3be5956b38271975039d7f2eccebb7ac0c7efe/appserver/payara-appserver-modules/microprofile/telemetry/src/main/java/fish/payara/microprofile/telemetry/tracing), so maybe there is something that could be learnt from it. There is a very similar error that was reported a little more than a year ago : [Bug Report: Payara 6.2023.11 - StatelessSessionBean with OpenTelemetry annotation @WithSpan throws jakarta.ejb.CreateException 路 Issue #6488 路 payara/Payara 路 GitHub](https://github.com/payara/Payara/issues/6488) but there hasn't had any update in a while. If there is somewhere a tutorial or a working example application that explains how to use `@WithSpan` in Payara, then I think it would be a good hint at what is wrong here. As an alternative, trying to achieve the same through configuration with `otel.instrumentation.methods.include`, or even avoiding capturing custom spans with this setup could be also tried to see if that works. ---
**Author:** ![montezott](https://avatars.discourse-cdn.com/v4/letter/m/45deac/32.png) [@montezott](https://discuss.elastic.co/u/montezott)\ **Post date:** [March 27, 2025, 10:02am UTC](https://discuss.elastic.co/t/problem-with-distributing-tracing-for-jakarta-10-and-payara-6/372920/9 "2025-03-27T10:02:17Z")
Hello, do you have any progress with the root cause and fix for the issue? Thanks ---
**Author:** ![Sylvain\_Juge](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylvain_juge/32/55521_2.png) [@Sylvain\_Juge](https://discuss.elastic.co/u/Sylvain_Juge)\ **Post date:** [March 27, 2025, 10:36am UTC](https://discuss.elastic.co/t/problem-with-distributing-tracing-for-jakarta-10-and-payara-6/372920/10 "2025-03-27T10:36:42Z")
Hi, Unfortunately there hasn't had any progress on this Payara issue, the seemingly related [Bug Report: Payara 6.2023.11 - StatelessSessionBean with OpenTelemetry annotation @WithSpan throws jakarta.ejb.CreateException 路 Issue #6488 路 payara/Payara 路 GitHub](https://github.com/payara/Payara/issues/6488) hasn't had any updates, and we don't have access to the FISH-8156 internal issue to see if there is any workaround. Reaching out to Payara support or adding a comment on the linked issue would probably be the best option as there isn't much we can do on our side to investigate further this Payara-only issue. As an alternative to annotations did you tried using `otel.instrumentation.methods.include` configuration option as an alternative as I previously suggested ? ---
**Author:** ![ciment](https://avatars.discourse-cdn.com/v4/letter/c/c37758/32.png) [@ciment](https://discuss.elastic.co/u/ciment)\ **Post date:** [February 5, 2026, 9:52am UTC](https://discuss.elastic.co/t/problem-with-distributing-tracing-for-jakarta-10-and-payara-6/372920/11 "2026-02-05T09:52:54Z")
when I added jvm property `-Delastic.apm.disable_instrumentations=opentelemetry` to domain.xml, finally it works perfectly. command: ${PAYARA\_DIR}/bin/asadmin --user ${ADMIN\_USER} --passwordfile=$PASSWORD\_FILE create-jvm-options -Delastic.apm.disable\_instrumentations=opentelemetry