Manually propagating a distributed tracing context from third party http client (.net)

[Nicolas_Rey]

Kibana version:
7.16.3 (Docker)

Elasticsearch version:
7.16.3 (Docker)

APM Server version:
7.16.3 (Docker)

APM Agent language and version:
.net

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
Is there a way when a third party http client (apollo graphql that itself create a transaction on APM) consume on of my controller to propagate automatically the transaction it has already spawn on APM?

I already use this successfully to propagate my transaction across many events pushed/pulled against a broker, but should I do the same into my controller where my transaction is automatically spawn, from let's say the transaction id that the client may put into the header then override the implicit transaction with a new one using this transactionId ?

Thank you!

[Nicolas_Rey]

Could you confirm that the client http has to supply both
http.request.headers.Elastic-Apm-Traceparent and http.request.headers.Traceparent in the headers (with the same value ?) to make the implicit transaction spawn in the .net agent using it ?

[Wolfgang_Ziegler]

Hi @Nicolas_Rey,

correct - the headers you identified are responsible for propagating the distributed trace context via HTTP.
The Elastic agents support the official W3C Trace Context specification, which means that traceparent is already sufficient.
For backward compatibility, the elastic-apm-traceparent is also still in use.

You can find more details about that here in our documentation.

I hope that helps,
Wolfgang

[Nicolas_Rey]

Perfectly clear @Wolfgang_Ziegler thank you!

We face the case where the client (Apollo GraphQL) provides an id in ( http.request.headers.Traceparent) that doesn't exist in APM, so our .net transaction initiate a transaction with it (as expected) but obviously it's not linked to any client calling transaction.

Is it possible if the possible behavior if the APM client (nodejs) is sampled ?
By the way, I think this theory is wrong because the initiated transaction by Apollo exists, it simply not supplied in http.request.headers.Traceparent, but it is in http.request.headers.Elastic-Apm-Traceparent... instead.

So to be clear, the transaction initiated by the client (Apollo GraphQL) is on APM with a trace.id
f7d58d08e640843d027dc597a3ba8a41, but the supplied headers to the .net application are not equals: one is the trace.id of the client, and the second one is a new id unknow on APM.
Here are the supplied ids :
http.request.headers.Elastic-Apm-Traceparent = 00-f7d58d08e640843d027dc597a3ba8a41-eb4a28da6f23a747-01
and
http.request.headers.Traceparent = 00-21caaf095d7a73309be4bbe6cd17262e-44185e788ca0c232-01

The first one is basically the trace.id of the client, the second one is the one used by the .net application receiving the http call to initiate it's own transaction (not correlated to the client thus).

Any idea ?

[Wolfgang_Ziegler]

Hi @Nicolas_Rey,

let me make sure that I understood the scenario:

• You are monitoring a GraphQL client with the Elastic Node.js agent?
• This client calls into a .NET web service.
• Fort this outgoing HTTP calls we see contradicting values for the trace id in the elastic-apm-traceparent and W3C traceparent?

Does that sum it up?

[Nicolas_Rey]

Yes absolutely!

Both the Apollo GraphQL and .net web services use the dedicated APM clients (nodejs and .net).
In the .net controller, we don't create the transaction manually, it's implicit due to the .net client.

Important sidenote: on 6 monitored calls from Apollo, 5 were in this case. Only one provided the same ids for both header parameter (and the correlation worked as expected for this one)

[Wolfgang_Ziegler]

Thanks for the confirmation @Nicolas_Rey,

let's rope in someone from the Node.js agent (@nodejsdev) since this is where my knowledge ends.

[trentm (Trent Mick)]

Hi @Nicolas_Rey, I work on the Node.js APM agent.

The code in the Node.js APM agent that sets the traceparent headers looks like this:

    if (traceparentStr) {
      setter(carrier, 'traceparent', traceparentStr)
      if (this._conf.useElasticTraceparentHeader) {
        setter(carrier, 'elastic-apm-traceparent', traceparentStr)
      }
    }

Some general questions:

1. What version of the Node.js APM agent are you using?
2. Do you know if you are setting the useElasticTraceparentHeader config var?
3. Are you able to get a trace log of the Node.js APM agent? (docs: Troubleshooting | APM Node.js Agent Reference [3.x] | Elastic) That might help clarify some details.

[Nicolas_Rey]

Hello @trentm , thank you for your help.

The developper working on the nodejs stack gave me some infos:

• version 3.37.0
• env var useElasticTraceparentHeader is not set, so it should take the default value (true if I'm right ?)
• he's digging on his side, and we'll activate the trace as soons as possible to provide them to you

[Nicolas_Rey]

Hello @trentm and @Wolfgang_Ziegler

We finally found the problem, nothing related to Elastic APM : we also use Dynatrace with the http headers trace enabled

As we supposed, the headers may be overridden by Dynatrace with ids from it's own context. When disabling the option, the behavior has gone.

Thanks for your time, I hope it could help people facing the same situation

[trentm (Trent Mick)]

@Nicolas_Rey Thanks for following up with the solution.