# Incompability between APM Agent and Micrometer Opentelemetry

**URL:** <https://discuss.elastic.co/t/incompability-between-apm-agent-and-micrometer-opentelemetry/365212>\
**Category:** APM\
**Tags:** java\
**Created:** [August 20, 2024, 3:24pm UTC](https://discuss.elastic.co/t/incompability-between-apm-agent-and-micrometer-opentelemetry/365212 "2024-08-20T15:24:09Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![dcolazin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dcolazin/32/99770_2.png) [@dcolazin](https://discuss.elastic.co/u/dcolazin)\
**Post date:** [August 20, 2024, 3:24pm UTC](https://discuss.elastic.co/t/incompability-between-apm-agent-and-micrometer-opentelemetry/365212/1 "2024-08-20T15:24:09Z")

</div>

**APM Agent language and version** : Java, 1.50.0

**Original install method (e.g. download page, yum, deb, from source, etc.) and version**: downloaded from Maven repository

**Is there anything special in your setup?** : usual Spring application with micrometer-tracing-bridge-otel

**Description of the problem including expected versus actual behavior. Please include screenshots (if relevant)**:

When a new Micrometer scope is opened, or a baggage is added, it is expected that the traceId/spanId/custom baggage is also put in the MDC. How the MDC is decorated is an implementation detail of the bridge between Micrometer and the underlying observability system: for example, everything works fine using the APM Agent, Micrometer and Brave.

When using Micrometer and Opentelemetry, a custom `io.opentelemetry.context.ContextStorage` is used (`io.micrometer.tracing.otel.bridge.EventPublishingContextWrapper$1`, source [here](https://github.com/micrometer-metrics/tracing/blob/f82941a6386529c945737da846d87dcf429564e4/micrometer-tracing-bridges/micrometer-tracing-bridge-otel/src/main/java/io/micrometer/tracing/otel/bridge/EventPublishingContextWrapper.java#L32), relevant method is `attach`); this `ContextStorage` has some logic, using Micrometer API, to publish events to registered listeners. Some of such listeners are responsible to update the MDC.

Yet, when also using the APM Agent, a different `ContextStorage` is used (`co.elastic.apm.agent.opentelemetry.context.OTelContextStorage`, source [here](https://github.com/elastic/apm-agent-java/blob/24b2b190933c7823a8ee3ed910a633868a68b7ce/apm-agent-plugins/apm-opentelemetry/apm-opentelemetry-plugin/src/main/java/co/elastic/apm/agent/opentelemetry/context/OTelContextStorage.java#L32)); this `ContextStorage` does not wrap existing `EventPublishingContextWrapper$1` so the events are never published.

Is there a way to use the APM Agent keeping also the Micrometer functionality?

**Steps to reproduce** :

1. Build a minimal Spring web application, using micrometer-tracing-bridge-otel, which logs the content of the MDC.
2. Run the application without the APM Agent and perform a HTTP request to the application; in the logs the spanId/traceId/custom baggage should be present.
3. Set up the Java agent as usual ([as here](https://www.elastic.co/guide/en/apm/agent/java/master/setup-javaagent.html)).
4. Run the application with the APM Agent and perform a HTTP request to the application; in the logs the spanId/traceId/custom baggage are not present.

---

<div class="post-metadata">

**Author:** ![Sylvain\_Juge](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylvain_juge/32/55521_2.png) [@Sylvain\_Juge](https://discuss.elastic.co/u/Sylvain_Juge)\
**Post date:** [August 20, 2024, 4:07pm UTC](https://discuss.elastic.co/t/incompability-between-apm-agent-and-micrometer-opentelemetry/365212/2 "2024-08-20T16:07:17Z")

</div>

Hi,

The APM agent implements an OpenTelemetry bridge so it maps most of the calls to the otel API to "translate" them into calls to APM.

However, using custom context storage with a wrapper is not supported, as the provided context storage will be replaced by the one in the agent.

Can you try doing the same with OpenTelemetry instrumentation agent ? the APM server endpoint supports the OTLP protocol, so if this works this would be a simple alternative. The otel agent is quite similar here and I know that he span/trace IDs are included in the logs that are captured and sent by the agent (I'm not 100% sure they are injected in the MDC though, so the file-based logs might remain identical).

---

<div class="post-metadata">

**Author:** ![dcolazin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dcolazin/32/99770_2.png) [@dcolazin](https://discuss.elastic.co/u/dcolazin)\
**Post date:** [August 30, 2024, 9:29am UTC](https://discuss.elastic.co/t/incompability-between-apm-agent-and-micrometer-opentelemetry/365212/3 "2024-08-30T09:29:11Z")

</div>

Thank you for the response, I will try with the other agent and report back.

---

<div class="post-metadata">

**Author:** ![dcolazin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dcolazin/32/99770_2.png) [@dcolazin](https://discuss.elastic.co/u/dcolazin)\
**Post date:** [September 3, 2024, 12:15pm UTC](https://discuss.elastic.co/t/incompability-between-apm-agent-and-micrometer-opentelemetry/365212/4 "2024-09-03T12:15:59Z")

</div>

Works as expected! Luckily, I'm interested in only traces and annotation instrumentation, so the migration should be straightforward. Thank you!
