What's the best practice for propagating cross cutting span attributes like a customer email?

[fuleow]

We have a small set of cross cutting span attributes such as the customer's email which would benefit from being propagated to the spans of all our upstream services.

A gateway service early in the request path already adds the attribute to the span and makes further upstream requests. We can search for transactions with this span attribute from the gateway's APM page, but we'd also like to do so from upstream services. For example the billing team might want to search for requests by that customer email directly from their service's APM dashboard rather than go through the gateway's APM dashboard every time.

OpenTelemetry has the concept of baggage which can propagate key value pairs to upstream services.

open-telemetry/opentelemetry-specification/blob/main/specification/overview.md#baggage-signal

Is there something similar for Elastic APM and if not what would be the best way to propagate these cross cutting span attributes?

[Eyal_Koren]

Hi and thanks for the question!

Unfortunately, the baggage API is not supported in our OpenTracing bridge, nor it is planned to be supported in our first take of the OpenTelemetry bridge (coming soon as an experimental feature). I assume we will get to that at some point, but it is not on our immediate plans.

@fuleow

A gateway service early in the request path already adds the attribute to the span

How is that? Is it by using our public API, or through our automatic header capturing?
Note that the latter means that we automatically capture headers from incoming HTTP requests and messages/records and add them to the Transaction context. This means that if you handle the propagation yourself by setting this info through HTTP/message header, it will be captured by the downstream agent and added to the downstream transaction.

[fuleow]

Thanks @Eyal_Koren we are using the public API to add spans. We'd like to be able to propagate the headers upstream and this sounds like it would be possible by propagating the information as headers instead of using baggage.

In the OpenTelemetry world I think this can be done by creating our own ContextPropagator and configuring the tracer to include it in addition to the W3C Trace Context Propagator.

Is there a way to do something similar with the Elastic APM Agent? Basically we need to read certain headers from incoming requests and add it to the span attributes. For outgoing requests we need to read the span attributes and inject these values into the outgoing request headers.

[Eyal_Koren]

@fuleow

Is there a way to do something similar with the Elastic APM Agent?

I'm afraid that until we support that through OTel (or other API), there is no straightforward way of doing the propagation through the agent. In any case, you would need to add the header to the outgoing request (or response).

@fuleow

Basically we need to read certain headers from incoming requests and add it to the span attributes. For outgoing requests we need to read the span attributes and inject these values into the outgoing request headers.

So basically, all you are missing currently in the API is a way to getLabel(), is that correct? Then at any time you can get the currentTransaction(): at the beginning of the request handling you would setLabel() to it and in the outgoing requests you would read the label from it. I am I missing something?

[fuleow]

@Eyal_Koren

So basically, all you are missing currently in the API is a way to getLabel() , is that correct? Then at any time you can get the currentTransaction() : at the beginning of the request handling you would setLabel() to it and in the outgoing requests you would read the label from it. I am I missing something?

Yes that should work. Since APM makes a distinction between transactions and spans maybe the most logical way to address this is if labels added to transactions can be propagated to upstream transactions.

For example the originating service would create a transaction with an email label

Transaction transaction = ElasticApm.startTransaction();
transaction.setLabel("email", "abc@example.com");

Upstreams would start their own transactions via ElasticApm.startTransactionWithRemoteParent and the email label from the parent transaction would automatically be added to the child transactions.

[Eyal_Koren]

Yes, what you propose would be idea, but it is a long term thing that would rely on the support of baggage, and would probably be implemented as a cross-agent solution.
If support through Java services is good enough for you, and you are willing to do the actual header send/read, then getLabel may be a quick solution (still not guaranteed how quick )