Spring Cloud Gateway and subsequent service has different trace id

Elastic Observability APM
1

I have added elastic apm with spring cloud gateway and two other microservices. I have added elastic-apm-agent as java agent for all the services. When I do a request which will go through gateway -> service A -> service B, the trace.id in the gateway services does not match with service A or service A. Please note that service A and service B has same trace id for the request which is expected. All the services are using webflux client netty. I have created a sample project with reproduction.

Please let me know whether the issue is a known issue or not. I have created a small app to reproduce the behaviour I am seeing. Here is the link. Please let me know if any additional information is needed.

Kibana version: 8.4.3

Elasticsearch version: 8.4.3

APM Server version: 8.4.3

APM Agent language and version: Java, 1.52.1

2

Could you try using the EDOT agent which is a quick check whether it's agent specific. You can just swap the jar and set the equivalent options

3

For EDOT agent, spring cloud gateway is generating the same trace id as the other services and working properly.

4

Thanks for the check and the app. I'll open an issue for the classic agent, but I recommend using the EDOT agent since that works and consequently we probably won't add the gateway support soon

5

Thank you for your quick response. I think I have found a workaround. My approach was using the apm-agent-api, I have manually added the header using ElasticApm.currentSpan().injectTraceHeaders(builder::header); in one of my filter. With the addition of header, now I am getting the same trace id in all the services for the single request. Please let me know if you see any negative consequence of my approach.