# Duplicate Traces in APM Observability after upgrade to kibana 8.16.5 from 8.16.2

**URL:** <https://discuss.elastic.co/t/duplicate-traces-in-apm-observability-after-upgrade-to-kibana-8-16-5-from-8-16-2/376712>\
**Category:** APM\
**Tags:** java, ui\
**Created:** [April 2, 2025, 8:07pm UTC](https://discuss.elastic.co/t/duplicate-traces-in-apm-observability-after-upgrade-to-kibana-8-16-5-from-8-16-2/376712 "2025-04-02T20:07:58Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![VVK](https://avatars.discourse-cdn.com/v4/letter/v/e47774/32.png) [@VVK](https://discuss.elastic.co/u/VVK)\
**Post date:** [April 2, 2025, 8:07pm UTC](https://discuss.elastic.co/t/duplicate-traces-in-apm-observability-after-upgrade-to-kibana-8-16-5-from-8-16-2/376712/1 "2025-04-02T20:07:58Z")

</div>

**Kibana version** :  
Kibana upgraded to 8.16.5 from 8.16.2 in self managed Observability  
**Elasticsearch version** :  
8.16.5  
**APM Server version** :  
apm-server:8.16.2  
**APM Agent language and version** :  
Java / elastic-apm.version: 1.52.1  
Programmatic attach: ElasticApmAttacher.attach();  
**Browser version** :  
Irrelevant

**Fresh install or upgraded from other version?**  
upgraded from 8.16.2 to 8.16.5

We have a self managed observability cluster test and prod. Recently kibana was upgraded from 8.16.2 to 8.16.5

It is really nice that we started getting significant details in trace after upgrade.  
However, We started getting a strange behaviour of getting duplicate entries in our observability APM traces.

As an example:  
Below images show "same call for multiple times'. The trace gives error message saying transaction was initially orphaned.  
The traceid section in 'TraceParent' header in each of Rest calls that were initiated by browser is same.  
For duplicate REST entries, it is observed that two headers are captured by tracer although at browser side only one is seen. These two headers are "Elastic-Apm-Traceparent" & "Traceparent"; where as for non duplicated trace rest calls, only one HTTP header Traceparent has been captured.

Below image show duplicate rest calls. However there were no duplicate calls from browser.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/8/5866693763f39821f149d099cf93812c15c0d3cb.png)

Any help in this would be great.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [April 2, 2025, 9:34pm UTC](https://discuss.elastic.co/t/duplicate-traces-in-apm-observability-after-upgrade-to-kibana-8-16-5-from-8-16-2/376712/2 "2025-04-02T21:34:18Z")

</div>

Hi @VVK

I think this issue has been identified and a Fix is coming in 8.19

> <https://github.com/elastic/kibana/issues/213074>
>
> The waterfall is rendering the same span document (with the same trace.id and sp…an.id) multiple times:
> 
> !\[Image\](https://github.com/user-attachments/assets/4192fee0-2cd6-412e-9981-3727c0689904)
> 
> !\[Image\](https://github.com/user-attachments/assets/c90a9646-c179-4eda-a04a-c70b25701df2)

> <https://github.com/elastic/kibana/pull/214957>
>
> \## Summary
> 
> Closes #213074
> 
> This PR prevents \`getChildrenGroupedByParentId\` …to include the parent item in the children list, as this was causing some duplication.
> 
> | Before | After |
> |-------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------|
> \<img width="1433" alt="image" src="https://github.com/user-attachments/assets/788684a8-21d7-48a6-820c-07b1fb3d0045" /\>|\<img width="858" alt="image" src="https://github.com/user-attachments/assets/b68129e1-137d-42fe-a7ce-70373447ece9" /\>|
> |\<img width="1372" alt="image" src="https://github.com/user-attachments/assets/ff6a5ac8-b46a-4eea-9c4c-638f4b479dc8" /\>|\<img width="844" alt="image" src="https://github.com/user-attachments/assets/31ef881c-a6d0-41ea-80d4-aebd587e76cd" /\>|

> <https://github.com/elastic/kibana/issues/212797>
>
> Currently in order to show the orphan items in the waterfall we use \[the entry t…ransaction as parent \](https://github.com/elastic/kibana/pull/210210/files#diff-4a044289408ff16fbe09c0b8656aa0ffc54508fcf5209d412ef521dc69921b26R463)as we need to link them to the waterfall - this can be confusing in some cases.
> 
> We should move the orphan spans to a new tree (every orphan should be the root transaction there)

---

<div class="post-metadata">

**Author:** ![VVK](https://avatars.discourse-cdn.com/v4/letter/v/e47774/32.png) [@VVK](https://discuss.elastic.co/u/VVK)\
**Post date:** [April 24, 2025, 7:14pm UTC](https://discuss.elastic.co/t/duplicate-traces-in-apm-observability-after-upgrade-to-kibana-8-16-5-from-8-16-2/376712/3 "2025-04-24T19:14:05Z")

</div>

@stephenb  
Has the fix been back-ported to 8.17?

I just wanted to get information from elastic instead of messages flying on internet.

Warm Regards,

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [April 24, 2025, 9:21pm UTC](https://discuss.elastic.co/t/duplicate-traces-in-apm-observability-after-upgrade-to-kibana-8-16-5-from-8-16-2/376712/4 "2025-04-24T21:21:37Z")

</div>

@VVK I encourage you to look at the Issues / Pull Request as they have the latest information

This one Looks like Backported to 8.17.5

> <https://github.com/elastic/kibana/pull/214957>
>
> \## Summary
> 
> Closes #213074
> 
> This PR prevents \`getChildrenGroupedByParentId\` …to include the parent item in the children list, as this was causing some duplication.
> 
> | Before | After |
> |-------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------|
> \<img width="1433" alt="image" src="https://github.com/user-attachments/assets/788684a8-21d7-48a6-820c-07b1fb3d0045" /\>|\<img width="858" alt="image" src="https://github.com/user-attachments/assets/b68129e1-137d-42fe-a7ce-70373447ece9" /\>|
> |\<img width="1372" alt="image" src="https://github.com/user-attachments/assets/ff6a5ac8-b46a-4eea-9c4c-638f4b479dc8" /\>|\<img width="844" alt="image" src="https://github.com/user-attachments/assets/31ef881c-a6d0-41ea-80d4-aebd587e76cd" /\>|

This one does not look to be backported only 8.19 forward

> <https://github.com/elastic/kibana/pull/214704>
>
> \## Summary
> 
> Closes #212797
> 
> 
> This PR filters out upstream orphans in the wa…terfall, which was confusing as we were reparenting to the entry transaction.

If I get a chance I will check why only one...

---

<div class="post-metadata">

**Author:** ![VVK](https://avatars.discourse-cdn.com/v4/letter/v/e47774/32.png) [@VVK](https://discuss.elastic.co/u/VVK)\
**Post date:** [April 25, 2025, 9:46am UTC](https://discuss.elastic.co/t/duplicate-traces-in-apm-observability-after-upgrade-to-kibana-8-16-5-from-8-16-2/376712/5 "2025-04-25T09:46:47Z")

</div>

thank you for your help

---

<div class="post-metadata">

**Author:** ![Damola\_Ajala](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/damola_ajala/32/139912_2.png) [@Damola\_Ajala](https://discuss.elastic.co/u/Damola_Ajala)\
**Post date:** [April 25, 2025, 10:10am UTC](https://discuss.elastic.co/t/duplicate-traces-in-apm-observability-after-upgrade-to-kibana-8-16-5-from-8-16-2/376712/6 "2025-04-25T10:10:44Z")

</div>

Hi @VVK

I guess u are making use of the Cloud-based.

Kindly share a guide on how you installed the APM agent on the monitoring application.

Thank you

---

<div class="post-metadata">

**Author:** ![VVK](https://avatars.discourse-cdn.com/v4/letter/v/e47774/32.png) [@VVK](https://discuss.elastic.co/u/VVK)\
**Post date:** [April 25, 2025, 11:52am UTC](https://discuss.elastic.co/t/duplicate-traces-in-apm-observability-after-upgrade-to-kibana-8-16-5-from-8-16-2/376712/7 "2025-04-25T11:52:33Z")

</div>

@Damola_Ajala

Yes. I am using self hosted environment (entire observability/monitoring stack) - in aws.

APM agent we picked from springboot 3.4 pom. We programmatically attach. elastic-apm.version is 1.52.1  
Applications are hosted as μ services in different cluster of aws. APM Server (jaeger) is hosted in the same env as that of application (and monitoring data is sent monitoring observability stack)

```auto
        <dependency>
            <groupId>co.elastic.apm</groupId>
            <artifactId>apm-agent-attach</artifactId>
            <version>${elastic-apm.version}</version>
        </dependency>

```

How do we attach?  
One set of developers use first line as ElasticApmAttacher.attach() in the main program of springboot application.  
Other set of developers put the attach in application listener (so as to gain those 8-12 seconds in probes) despite the best practice that is documented in formal documentation.

```Java
@EnableMongoRepositories
@SpringBootApplication
public class MyServiceApplication { 
    public static void main( String[] args ) {
        ElasticApmAttacher.attach();
        SpringApplication.run(MyServiceApplication.class, args);
    }
} 

```

Regards

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [April 25, 2025, 2:34pm UTC](https://discuss.elastic.co/t/duplicate-traces-in-apm-observability-after-upgrade-to-kibana-8-16-5-from-8-16-2/376712/8 "2025-04-25T14:34:10Z")

</div>

@VVK  
Never Hurts To ask... this one (the other PR) is not backported to 8.17.6

> <https://github.com/elastic/kibana/pull/214704>
>
> \## Summary
> 
> Closes #212797
> 
> 
> This PR filters out upstream orphans in the wa…terfall, which was confusing as we were reparenting to the entry transaction.