This can happen when not properly deactivating a previous span or context

Elastic Observability APM
,
1

Current stack setup:

  1. Spring boot version: 3.4.1
  2. Apache camel version: 4.9.0
  3. ActiveMQ: 5.14.0
  4. Elastic apm agent version: 1.53.0
  5. Apm server version: 8.12.2
  6. Java version: 17

After upgrading Spring boot version from 2.7.6 to 3.4.1 and apache camel from 3.14.6 to 4.9.0 we partially lost trace id information in log messages. For some messages it logged empty trace id.

Before upgrade we had apache camel camel-opentracing-starter dependecy which worked great in cooperation with elastic-apm-agent-1.48.1.jar and APM server with version 8.12.2. But during apache camel upgrade from v3 to v4 camel-opentracing become deprecated and apache camel suggested to start using camel-opentelemetry-starter.

After upgrade we partialy lost trace id information in part of log messages. It logged empty trace id and some warning information like this:

2025-05-12 09:38:21.533 | 2025-05-12 06:38:21,532 [Camel (camel-1) thread #28 - JmsConsumer[annual.party.renewal]] WARN  co.elastic.apm.agent.opentelemetry.tracing.OTelSpan - The addEvent API is not supported at the moment
2025-05-12 09:38:21.538 | [2025-05-12 06:38:21,538][INFO ][annual-party-renewal-request-handle] {trace.id=90c5220d6c2e3b4b2dfb9ad7679f4ee6, transaction.id=311045dd12ac947a} Processing Party Renewal request
2025-05-12 09:38:21.544 | 2025-05-12 06:38:21,543 [Camel (camel-1) thread #28 - JmsConsumer[annual.party.renewal]] WARN  co.elastic.apm.agent.impl.ActiveStack - Deactivating a context (OTelBridgeContext[{opentelemetry-trace-span-key=OtelSpan['activemq' 00-90c5220d6c2e3b4b2dfb9ad7679f4ee6-5a49f3b9b8fc9bc5-01 (4d9154a7)]}]) which is not the currently active one ('JMS RECEIVE from queue annual.party.renewal' 00-90c5220d6c2e3b4b2dfb9ad7679f4ee6-311045dd12ac947a-01 (742172c3)). This can happen when not properly deactivating a previous span or context.
2025-05-12 09:38:22.861 | 2025-05-12 06:38:22,861 [Camel (camel-1) thread #28 - JmsConsumer[annual.party.renewal]] WARN  co.elastic.apm.agent.impl.ActiveStack - Deactivating a context (OTelBridgeContext[{opentelemetry-trace-span-key=OtelSpan['activemq' 00-90c5220d6c2e3b4b2dfb9ad7679f4ee6-5a49f3b9b8fc9bc5-01 (4d9154a7)]}]) which is not the currently active one (OTelBridgeContext[{}]). This can happen when not properly deactivating a previous span or context.
2025-05-12 09:38:23.879 | 2025-05-12 06:38:23,878 [Camel (camel-1) thread #58 - JmsConsumer[annual.policy.lifecycle.handle]] WARN  co.elastic.apm.agent.impl.ActiveStack - Deactivating a context (OTelBridgeContext[{opentelemetry-trace-span-key=OtelSpan['activemq' 00-90c5220d6c2e3b4b2dfb9ad7679f4ee6-de91df46cd8b6fbb-01 (372fca52)]}]) which is not the currently active one ('JMS RECEIVE from queue annual.policy.lifecycle.handle' 00-90c5220d6c2e3b4b2dfb9ad7679f4ee6-ddd3f4d93dc45faa-01 (7dd3b0d7)). This can happen when not properly deactivating a previous span or context.
2025-05-12 09:38:23.893 | 2025-05-12 06:38:23,892 [Camel (camel-1) thread #58 - JmsConsumer[annual.policy.lifecycle.handle]] WARN  co.elastic.apm.agent.impl.ActiveStack - Deactivating a context (OTelBridgeContext[{opentelemetry-trace-span-key=OtelSpan['activemq' 00-90c5220d6c2e3b4b2dfb9ad7679f4ee6-de91df46cd8b6fbb-01 (372fca52)]}]) which is not the currently active one (OTelBridgeContext[{}]). This can happen when not properly deactivating a previous span or context.
2025-05-12 09:38:23.901 | [2025-05-12 06:38:23,901][INFO ][annual-renewal-handle-serve] {} Processing Enrich request
2025-05-12 09:38:30.172 | [2025-05-12 06:38:30,171][INFO ][com.lv.app.enrichment.Service] {} Found bad debt score

Replacing elastic-apm-agent-1.53.0.jar java agent with opentelemetry-javaagent.jar trace id again were presented in log messages.

Is this some OpenTelemetry bridge problem?
We are not using manual instrumentation.
We are using apache camel provided component camel-opentelemetry in order to trace routes.

2

Did the log messages with empty trace IDs correspond to each "addEvent API is not supported"?

Are there specific features of the Elastic APM Java agent you need, or can you switch to using the EDOT Java agent?

3 
Are there specific features of the Elastic APM Java agent you need, or can you switch to using the EDOT Java agent?

I tried elastic-otel-javaagent-1.4.1.jar and trace ID were presented in log messages, but had the problem with metrics. Metric informations were missing in Elasticsearch (Kibana -> Observability -> Applications -> Service Inventory -> some app -> Metrics)
Tried to pass -Dotel.instrumentation.micrometer.enabled=true but without success.

Also didn`t find the way how to pass in elastic-otel-javaagent those parameters:
-Delastic.apm.service_node_name="service node"
-Delastic.apm.hostname="hostname"
-Delastic.apm.application_packages="application package"

Can it be passed via OTEL_RESOURCE_ATTRIBUTES?

I guess there are no such options in OpenTelemetry?
Have to check if there are specific features of the Elastic APM Java agent which we need.

Did the log messages with empty trace IDs correspond to each "addEvent API is not supported"?

Not sure. I made test for concrete use case and a log example was added here.

4

-Dotel.resource.attributes=host.name=somethingelse,...

application_packages is not needed

I'm not sure about the service node name, but I don't think you need to set it, it should find the right ID for that

micrometer should be enabled with that flag, what metric are you expecting that isn't present?

5 
micrometer should be enabled with that flag, what metric are you expecting that isn't present?

Sorry, my fault, had to wait a little bit longer. Now metrics are presented in Kibana. Only need to find out how to change from dot chart to line chart in tables.

image
image2257×869 112 KB

I'm not sure about the service node name, but I don't think you need to set it, it should find the right ID for that

It would be great if I could somehow manage to pass this service_node_name information, otherwise it does not look readable:

image
image2270×374 18.8 KB

So far everything looks good with elastic-otel-javaagent, except this "service_node_name" :slight_smile:

6

-Dotel.resource.attributes=host.name=else,service.instance.id=something

1 Like
7

A huge thank to You! service.instace.id=something works fine.
Migrating elastic-apm-agent to elastic-otel-javaagent solved the problem.

But returning to the original problem, do you have any clue why elastic-apm-agent + camel-opentelemtry does not work out of the box as it should?

8

There could be many different things. When there's an upgrade and especially where they've removed support for a technology it runs across instrumentation, the bridge, configuration. Given there's a valid solution, I won't be able to prioritizing spending more effort on this at this time