Inject more data to MDC, i.e. user.id

[Nadrendion (Johnny Nilsson)]

Hi,
I have a use case where I have a set of microservices deployed. All of them have the Elastic APM agent attached and log correlation is enabled successfully. I can get Logback to log the transaction.id and the other ids just fine.
The applications are written in java 17 with spring boot 3.

One application receives the incomming request, and then it will call the others depending on what the user wants to do.

Now... I would like to have more data injected into MDC, such as the user.id, or perhaps some custom property, i.e customer.id.

Is it somehow possible to set a value with the public api in the first application, and then have the apm-agent inject the value into the MDC?
How about also forwarding this information to other applications within the same transaction and also have it injected in their MDC?

If possible, how do I configure this functionality?

[Jonas_Kunz]

Hi @Nadrendion ,

I'll answer your question in two parts.

Is it somehow possible to set a value with the public api in the first application, and then have the apm-agent inject the value into the MDC?

The apm-agent does not inject anything except the standard trace correlation properties into the MDC.
You'll need to update the MDC yourself in your code, here is an example for SLf4J:

try(var closeable = MDC.putCloseable("user_id", "value")) {
     //every log in here will see the "user_id" MDC value
}

How about also forwarding this information to other applications within the same transaction and also have it injected in their MDC?

The elastic apm agent 1.41.0 added OpenTelemetry baggage support. Baggage is exactly the solution for your usecase: Propagating custom data across services.

You can use baggage using the OpenTelemetry API. The agent will take care of propagating the values via headers of external calls and take care of parsing the values on the receiving side.

For the sending side you could use the following snippet:

Baggage updated = Baggage.current().toBuilder()
        .put("user_id", "<user_id_here>")
        .build();
try (var scope = updated.makeCurrent()) {
        //every external call (e.g. http) in this block will propagate user_id to downstream services
}

On the downstream service you can then read the baggage value and put it in the MDC:

String user_id = Baggage.current().getEntryValue("user_id")
try(var closeable = MDC.putCloseable("user_id", user_id)) {
    //every log in here will see the "user_id" MDC value
}

With the upcoming release the agent will automatically add the baggage contents to your spans and transactions as labels.

[Nadrendion (Johnny Nilsson)]

Thank you!
I will try to build a solution based on your input on monday.
I will be using a OncePerRequestFilter and do the magic from there.

Btw, the Baggage-parameters, are they searchable in elastics or would I need to use setUser for that?

[Jonas_Kunz]

Btw, the Baggage-parameters, are they searchable in elastics or would I need to use setUser for that?

In version 1.41.0 of the agent the baggage is not automatically written to your span/transaction documents. You'll have to add them manually via the API.

With the next release there will be a new baggage_to_attach config option to automatically add baggage entries to the spans/transactions.

[Nadrendion (Johnny Nilsson)]

I have implemented this successfully. Thank you!

I, however, may have discovered a possibility for improvement. Not sure how this should be reported.

If an application launches new threads for concurrent tasks to be executed, then the APM-agent injects the trace, transaction and error-ids properly, but I can't see how I can plugin a method for also transferring my custom values between threads.

[Jonas_Kunz]

I assume the problem you are describing is that in the new thread the value is not automatically added to the MDC?

After you've added your value to the baggage, it should be automatically propagated to new threads just like the trace-ID. The problem is that even though the value will be in the baggage on the new Thread, it won't be in the MDC because of the thread switch.

The only solution I can provide you here is to manually wrap your Executors so that they wrap submitted Runnables to read the current value from the baggage and put it into the MDC of the new thread.