How to obtain an OpenTelemetry reference to use to manually instrument code?

Elastic Observability APM
1

Hello, I'm using 1.36.0 of the apm java agent against 7.17.9 apm server.

My app has manually instrumented code that I want to integrate with the apm traces.

Previously when using opentracing, I did this via just making a new ElasticApmTracer and it all worked.

Now I am using OpenTelemetry, everything seems to have gone very wrong! Even if in my instrumentation I use a noop tracer, it somehow seems to interfere with the apm agents traces, and has the consequence that while all the spans get reported same as they use to, nothing gets properly linked up as a transaction. I assume my noop tracer is somehow messing up apm spans.

I've looked through GitHub - elastic/apm-agent-java: Elastic APM Java Agent and really can't work out how to get a valid OpenTelemetry instance.

I see ElasticOpenTelemetry.java, but it requires a co.elastic.apm.agent.impl.ElasticApmTracer which is internal and in any case looks tricky to instantiate.

I see also GlobalOpenTelemetryInstrumentation.GlobalOpenTelemetryAdvice.ELASTIC_OPEN_TELEMETRY which returns an OpenTelemetry for me, and does look like the right sort of thing, but it's in apm-opentelemetry-plugin which sounds like a wrong thing for my library code to depend on, and I can't find any reference to ELASTIC_OPEN_TELEMETRY in documentation so I'm not sure I'm supposed to use it.

Based on OpenTelemetry bridge | APM Java Agent Reference [1.x] | Elastic I tried just doing GlobalOpenTelemetry.get(), but it didn't work either.

What is the correct way to obtain an OpenTelemetry instance I can use to integrate with elastic apm agent? I need an OpenTelemetry instance specifically because my code is polymorphic across lots of different tracing implementations.

I'd actually be happy at present just if the apm traces would work even ignoring my manual instrumentation, I find this behaviour with the nooptracer from opentelemetry messing up apm transactions very surprising.

My code using 1.22.0 dependencies of opentelemetry, maybe a conflict with 1.0.0 opentelemetry is to blame for all this strange behaviour... bit hard to image.

I don't open or close any scopes so I really can't see how the nooptracer is interact with apm tracing

2

Hi and welcome to our forum! :wave:

What doesn't work exactly?
Please note that the agent should be attached just the same as it was before (not as a dependency). Then, the only dependency you should add to your app is opentelemetry-api.
Note that we continuously run tests with 1.22.0 as well, so my guess it is only a setup issue.

My code using 1.22.0 dependencies of opentelemetry, maybe a conflict with 1.0.0 opentelemetry is to blame for all this strange behaviour... bit hard to image.

What do you mean by that? What other OTel dependency to you have? If you use our agent and want to use the OTel API for manual tracing, then the OTel bridge is indeed the right thing, in which case you need only one dependency. Alternatively, you can use the OTel agent and still send traces to the Elastic stack, but in this case you shouldn't use both Elastic and OTel APM agents.

I hope this helps.
If not, please share the full agent and app settings, as well as the code which doesn't work as expected.

3

Hey, thank you for your response. I've got a bit further now and understand the issue a lot better.

GlobalOpenTelemetry.get() does work, sorry. The issue I am having actually is this.

Previously I used an opentracing grpc interceptor in my code and disabled apm grpc instrumentation.

Now that I am using opentelemetry, I am using the grpc instrumentation that opentelemetry provide which as detailed here OpenTelemetry bridge | APM Java Agent Reference [1.x] | Elastic apm doesn't interact with.

That's fine, so I made it so that when the apm agent is in use, I don't have this grpc instrumentation in use and instead use the apm grpc instrumentation.

My code basically looks like

rest api -> grpc client call -> grpc service handler -> actual logic

Previously when I used opentracing this would all come through as one transaction.

When I use opentelemetry, disable my grpc instrumentation and enable apms grpc instrumentation then I get 3 transactions, a rest api transaction with no spans linked to it, same with the grpc client call and same with the grpc service handler call. The actual logic spans are put into elastic, but not linked to any transaction.

4

Please go over my comment above and see if you realize what can be the issue, including improper dependencies, or provide more info as I asked for, showing your agent configuration, application configuration, dependencies etc. Setting the agent's log_level=debug may provide more useful info.

Note that while OpenTracing was only an API, with libraries providing only calls to it, while in OTel there is an API as well as an implementation, and I don't know what and how you are using.
If you are using two APM agent implementations (Elastic and OTel) together, then the resulting traces are unpredictable.

Also, why do you disable the agent's gRPC instrumentation? What are you not getting with it?

5

What doesn't work exactly?

The thing that doesn't work is that the spans for a request are not getting linked up in a single transaction. They previously were when I used an opentracing grpc interceptor.

Please note that the agent should be attached just the same as it was before (not as a dependency). Then, the only dependency you should add to your app is opentelemetry-api.
Note that we continuously run tests with 1.22.0 as well, so my guess it is only a setup issue.

Yep correct, I was wrong to suspect a dependency issue. I don't have the agent as a dependency.

What do you mean by that? What other OTel dependency to you have? If you use our agent and want to use the OTel API for manual tracing, then the OTel bridge is indeed the right thing, in which case you need only one dependency. Alternatively, you can use the OTel agent and still send traces to the Elastic stack, but in this case you shouldn't use both Elastic and OTel APM agents.

I use several otel dependencies, but they're mostly unimportant. For instance I also have a otel dependency that pertains to jaeger, but that class will only be loaded if the user has specified they are using jaeger for tracing.

If not, please share the full agent and app settings, as well as the code which doesn't work as expected.

The full agent is very simple, I do

-javaagent:apmagent.jar -Delastic.apm.server_urls=someurl -Delastic.apm.service_name=somename -Delastic.apm.config_file:./elasticapm.properties

elasticapm.properties is also very simple, I do

application_packages=myrootpackage
log_level=INFO

It's not possible for me to post the code which doesn't work as expected, really, because what it is is a long sequence of code, which goes rest api -> grpc client -> grpc service -> other logic

And the issue I am seeing is there are independent transactions for rest api and grpc service, and no transaction linking them up, and these transactions don't have any spans associated with them.

Please go over my comment above and see if you realize what can be the issue, including improper dependencies, or provide more info as I asked for, showing your agent configuration, application configuration, dependencies etc. Setting the agent's log_level=debug may provide more useful info.

You'll have to trust me a bit on this, I did set the log_level to trace, and actually this helped identify lots of issues, but now the logs look very clean.

Note that while OpenTracing was only an API, with libraries providing only calls to it, while in OTel there is an API as well as an implementation, and I don't know what and how you are using.
If you are using two APM agent implementations (Elastic and OTel) together, then the resulting traces are unpredictable.

I'm not doing this, any more, now I am just using apm agent and GlobalOpenTelemetry.get()

Also, why do you disable the agent's gRPC instrumentation? What are you not getting with it?

It wasn't my idea really, and actually now I have reinstated it. The thing is the tracing code that exists in the application seeks to generically support many tracing implementations, some of which are entirely manually instrumented and don't involve a javaagent.

And it had unconditional grpc instrumentation. If the apm grpc instrumentation was also enabled, this caused conflicts with our isntrumentation in that our instrumentation was trying to open and close context (by which I mean, calling span.makeCurrent() and later down the line closing the scope. I have since made it so that this grpc instrumentation is conditionally applied, and in the case of apm it is not applied.

In fact, I have disabled all manual instrumentation of my code in the case of apm

6

Sorry, I am having trouble fully understanding your setup, so I will just try to help with some general info and a concrete advice.

At the end, there is no APM tracing without an agent at all. Tracing "implementations" may mean different things to different people, so just to align on concepts, I'll explain the terminology I am using:
In order to get APM traces, you needs to have two sets of capabilities:

  1. hook into libraries/interfaces/custom-code and send signals to an agent core that mark the start and end or a traced event, like a span. We refer to this set of capabilities collectively as "instrumentation".
  2. an agent "core" that receives these events and takes care of all the rest: creating and managing the objects that represent the events, collect metrics, serialize and send events to the backend system etc.

If I understand correctly, for (2) you are using only Elastic APM agent and not the OTel agent. Either way, you should use only one that listens to events sent by the hooks in through the same API (1).
For (1) you can either rely on hooks installed by the agent (through bytecode instrumentation) or you can manually place them, in which case you can communicate with the core through a known API. For that, you are relying on the OTel API, which is good. When Elastic APM Agent is installed, it will automatically register to listen on events being triggered by the instrumentation hooks. Internally, the Elastic APM agent employs a "bridge" so that events created through manual instrumentation can be incorporated into active traces, either if they are created by the agent's instrumentation or manual instrumentation.

And now to the concrete advice: if you don't want to rely on the traces that are initiated by the Elastic APM agent's inherent gRPC instrumentation- that is fine, you can disable it and replace it by manual one like I think you did. However, once you do that, it is your responsibility to make sure that the tracing context is properly transferred within the thread, between threads of the process and through communication to different processes. For example, in order to have nested spans on the same thread through the OTel API, you need to properly use the makeCurrent() API, otherwise current() on the same thread won't work. The context propagation between threads on the same JVM is handled by the Elastic APM agent, as long as your app is relying on supported async execution frameworks.
If you disable client spans created by the Elastic APM agent as well (which is probably the case if you disable gRPC instrumentation altogether), then you must take care of inter-process context propagation as well. In OTel API, this is done through Propagators.

I hope this helps.
Good luck

7

Thank you very much for the help. I (genuinely) realise that I haven't asked my question very well and I'm sorry about that.

You've helped clarify several things, and I do now understand what I need to do to solve my problem. Namely I need to extract the relevant parts of the code in the app that I work on and create a miniature verson of the problem that I can study.

I will report back once I've resolved my issue to help anyone else who has the same sort of issue I'm experiencing.

Thanks again.

1 Like
8

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.