Incompability between APM Agent and Micrometer Opentelemetry

Elastic Observability APM
1

APM Agent language and version: Java, 1.50.0

Original install method (e.g. download page, yum, deb, from source, etc.) and version: downloaded from Maven repository

Is there anything special in your setup?: usual Spring application with micrometer-tracing-bridge-otel

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

When a new Micrometer scope is opened, or a baggage is added, it is expected that the traceId/spanId/custom baggage is also put in the MDC. How the MDC is decorated is an implementation detail of the bridge between Micrometer and the underlying observability system: for example, everything works fine using the APM Agent, Micrometer and Brave.

When using Micrometer and Opentelemetry, a custom io.opentelemetry.context.ContextStorage is used (io.micrometer.tracing.otel.bridge.EventPublishingContextWrapper$1, source here, relevant method is attach); this ContextStorage has some logic, using Micrometer API, to publish events to registered listeners. Some of such listeners are responsible to update the MDC.

Yet, when also using the APM Agent, a different ContextStorage is used (co.elastic.apm.agent.opentelemetry.context.OTelContextStorage, source here); this ContextStorage does not wrap existing EventPublishingContextWrapper$1 so the events are never published.

Is there a way to use the APM Agent keeping also the Micrometer functionality?

Steps to reproduce:

  1. Build a minimal Spring web application, using micrometer-tracing-bridge-otel, which logs the content of the MDC.
  2. Run the application without the APM Agent and perform a HTTP request to the application; in the logs the spanId/traceId/custom baggage should be present.
  3. Set up the Java agent as usual (as here).
  4. Run the application with the APM Agent and perform a HTTP request to the application; in the logs the spanId/traceId/custom baggage are not present.
2

Hi,

The APM agent implements an OpenTelemetry bridge so it maps most of the calls to the otel API to "translate" them into calls to APM.

However, using custom context storage with a wrapper is not supported, as the provided context storage will be replaced by the one in the agent.

Can you try doing the same with OpenTelemetry instrumentation agent ? the APM server endpoint supports the OTLP protocol, so if this works this would be a simple alternative. The otel agent is quite similar here and I know that he span/trace IDs are included in the logs that are captured and sent by the agent (I'm not 100% sure they are injected in the MDC though, so the file-based logs might remain identical).

3

Thank you for the response, I will try with the other agent and report back.

4

Works as expected! Luckily, I'm interested in only traces and annotation instrumentation, so the migration should be straightforward. Thank you!