Elastic APM + Yarp

[Thomas_Piart]

Hello,

I'm doing an http call to server, which in turn does an http call to another server

All that is instrumented with Elastic APM

Http > Server A > Http > Server B

If I use an httpclient to do the call from server A to server B => I see the full result in Elastic APM

If I use Yarp (a .Net reverse proxy) to do the same call, I don't see the full result in Elastic APM
I see both call separated, but not "linked" in the UI

A similar request has already been posted: elastic-apm-dotnet-yarp-issue/343247

But with this one, I provide a gthub repo to reproduce:

GitHub - tomap/repro-yarp-elastic-apm: repo to reproduce issue between...

1/ Launch both projects
2/ call /getme from WebApplication2 => this works fine (using httpclient in a controller)
3/ call /weatherforcast => does not work fine (only part of the trace visible in Elastic APM)

I'm wondering if the X-Forwarded headers added by yarp are messing with Elastic APM collection

I've used a free Elastic cloud trial to repro

I hope you have access to the data from my trial account:
4ad8fed546814708965668fdd56b1d56 us-central1

TIP 2: Check out the troubleshooting guide first. Not only will it help you to resolve common problems faster but it also explains in more detail which information we need before we can properly help you.

APM Agent language and version: 1.25.1 / dotnet 7

Original install method (e.g. download page, yum, deb, from source, etc.) and version:

[kuanpak (Jacky Chu)]

I had the similar issue before, it should be related to Yarp customized DistributedContextPropagator. Moreover, if the incoming requests to Yarp contain traceparent / tracestate headers, it will capture the wrong parent-child hierarchy as Elastic APM does always resue the proxied HttpClient trace context headers if exists.

Here is the workaround you can apply in Yarp app to avoid those issues:

builder.Services.AddReverseProxy()
    .AddTransforms(context => context
        .AddRequestHeaderRemove("traceparent")
        .AddRequestHeaderRemove("tracestate"))
    .LoadFromConfig(builder.Configuration.GetSection("ReverseProxy"))
    .ConfigureHttpClient((context, handler) => handler.ActivityHeadersPropagator = DistributedContextPropagator.CreateDefaultPropagator())
    ;