Duplicate Traces in APM Observability after upgrade to kibana 8.16.5 from 8.16.2

Elastic Observability APM
,
1

Kibana version:
Kibana upgraded to 8.16.5 from 8.16.2 in self managed Observability
Elasticsearch version:
8.16.5
APM Server version:
apm-server:8.16.2
APM Agent language and version:
Java / elastic-apm.version: 1.52.1
Programmatic attach: ElasticApmAttacher.attach();
Browser version:
Irrelevant

Fresh install or upgraded from other version?
upgraded from 8.16.2 to 8.16.5

We have a self managed observability cluster test and prod. Recently kibana was upgraded from 8.16.2 to 8.16.5

It is really nice that we started getting significant details in trace after upgrade.
However, We started getting a strange behaviour of getting duplicate entries in our observability APM traces.

As an example:
Below images show "same call for multiple times'. The trace gives error message saying transaction was initially orphaned.
The traceid section in 'TraceParent' header in each of Rest calls that were initiated by browser is same.
For duplicate REST entries, it is observed that two headers are captured by tracer although at browser side only one is seen. These two headers are "Elastic-Apm-Traceparent" & "Traceparent"; where as for non duplicated trace rest calls, only one HTTP header Traceparent has been captured.

Below image show duplicate rest calls. However there were no duplicate calls from browser.

image
image1848×541 81.7 KB

Any help in this would be great.

2

Hi @VVK

I think this issue has been identified and a Fix is coming in 8.19

3

@stephenb
Has the fix been back-ported to 8.17?

I just wanted to get information from elastic instead of messages flying on internet.

Warm Regards,

4

@VVK I encourage you to look at the Issues / Pull Request as they have the latest information

This one Looks like Backported to 8.17.5

This one does not look to be backported only 8.19 forward

If I get a chance I will check why only one...

5

thank you for your help

6

Hi @VVK

I guess u are making use of the Cloud-based.

Kindly share a guide on how you installed the APM agent on the monitoring application.

Thank you

7

@Damola_Ajala

Yes. I am using self hosted environment (entire observability/monitoring stack) - in aws.

APM agent we picked from springboot 3.4 pom. We programmatically attach. elastic-apm.version is 1.52.1
Applications are hosted as μ services in different cluster of aws. APM Server (jaeger) is hosted in the same env as that of application (and monitoring data is sent monitoring observability stack)

        <dependency>
            <groupId>co.elastic.apm</groupId>
            <artifactId>apm-agent-attach</artifactId>
            <version>${elastic-apm.version}</version>
        </dependency>

How do we attach?
One set of developers use first line as ElasticApmAttacher.attach() in the main program of springboot application.
Other set of developers put the attach in application listener (so as to gain those 8-12 seconds in probes) despite the best practice that is documented in formal documentation.

@EnableMongoRepositories
@SpringBootApplication
public class MyServiceApplication { 
    public static void main( String[] args ) {
        ElasticApmAttacher.attach();
        SpringApplication.run(MyServiceApplication.class, args);
    }
}

Regards

8

@VVK
Never Hurts To ask... this one (the other PR) is not backported to 8.17.6