Any plans for manual instrumentation (java)?

Elastic Observability APM
1

Hi,
I read through the supported technologies page, and for us there is a lot of things missing to even get started, we have a variety of "weird" old and new libs for external communication and lots of different systems that use different approaches.

I also found the APM server "Intake API" which sounds interesting. Is there any thoughts or plans to have some kind of manual (code) instrumentation support that uses this API?

Regards /Johan

2

Hi Johan,

Thanks for considering our APM solution.
I believe what you are looking for is our public API, which enables you to trace any part of your code and incorporate it with the native agent tracing.
Alternatively, you can use our OpenTracing bridge if you are more comfortable with the OpenTracing API.
Both still limit you to trace only your own code, though.

We currently don't provide a way to instrument libraries, but we intend to add that in the future.

I hope this helps and will be glad to get any feedback.

Thanks,
Eyal.

3

I believe that is what I am looking for. Will try it out!

Thanks /Johan

4

Hi Eyal,

I've used the public API and annotations for Java and they work quite well in most situations.
The only issue for myself is that annotations could still be improved as I've discussed in a different thread here: Elastic APM Java Annotation API Felix provided a workaround but we can't use that workaround all the time; still better than nothing :slight_smile:

One thing I couldn't find a reference to is if it is possible to manually set the traceID and/or transactionID of a span or trace. The use case for this is to be able to propagate IDs across services that do not yet have Elastic APM agents.

Let us say we have a Java service, we can get the current transactionID and embed it in the message payload as a key-value pair, pass the message to a legacy Perl service, which process it and then passes it to another Java service. The second Java service then sets the parent ID to the embedded transaction ID.

I've looked at the OpenTracing bridge but I'm not sure it can really be used in this case.

Thanks,

Ronald

5

HI again,
Just a few questions.

  1. It seems like I do not have the possibility to do ElasticApm.currentOrStartTransaction(), which might be a problem since then we have to know in which context we are. Some things run with tomcat, spring and some use vertx. I have not thought about this 100% but it seems like it could be an issue. Is it possible to disable the agent from creating transactions and leave it to be completely manual?

Found it: Manual instrumentation If you don’t want Elastic APM to auto-instrument known frameworks, but instead only rely on manual instrumentation, disable the auto instrumentation setting the configuration option instrument to false .

  1. I can also see that it depends on thread-local to manage transactions and spans. Is there any ideas or best-practices for usage in async environments like vertx, rx or flux? We are trying to move away from one-request-per-thread model and do not want to be limited by our tracing solution.

Thanks /Johan

6

Ronald,

Thanks for reaching out, though this better go in a separate thread.

Why can't you use the OpenTracing OpenTracing inject-extract API? You can use inject in the caller Java service, then produce an arbitrary header in your Perl service that adheres to the header format, pass it through to the second Java service and extract it there. Can't this work?

BTW, we have a plan to add something similar to the OpenTracing inject-extract API to our public API (see in https://github.com/elastic/apm-agent-java/issues/334).

I hope this makes sense. In any case, let's continue this conversation to a different thread.

Thanks,
Eyal.

7

Hi back again :slight_smile:

I think 1 and 2 are related to each other- it is about the basic tracing of async executions.
We are planning to address all kind of async scenarios in the near future, so we will have a reference implementation, but you may still use the API to monitor such.

Essentially, a Transaction is not reflecting something running on a thread, but an incoming request. When using all APIs besides the annotations API, a span/transaction will be attached to the thread only if you activate them explicitly.
If your code is forking a task to a different thread, your manual instrumentation could create a span in the initiating thread, then pass the span to the forked thread, where you can activate it and use it. This should allow you monitoring async scenarios.
Just make sure to follow instructions about closing the scope (try-with-resource is recommended), and to add data to each span only when you where you can control its lifecycle (meaning - when you are certain it is not ended already).

If all this makes sense, rethink shutting down the auto instrumentation, maybe you don't need to after all.

I hope it DOES make sense.

Good luck,
Eyal.

8

Hi,

From the docs, I did not understand that what was happening with activate, may have to read it again.
I will dig into it, the only way to understand how it works.

Thanks!

9

Hi Eyal,

Thanks for the response. I think I missed to mention that the Perl service is a black box that we cannot do any instrumentation on so there is no way for us to propagate via OpenTracing. However, it does support arbitrary data payloads in which we can embed trace and span and transaction IDs as needed. We have a number of services like that in which the only way for us to propagate IDs is to embed these in the data payload so we can extract and reuse in services we can instrument.

That link you gave would seem to give us that capability. Looking forward to it being available soon :slight_smile:

Edit:
I looked at the OpenTracing bridge again and re-studied the inject-extract in OpenTracing and think we can use the approach you mentioned. We'll try it out.

Thanks!

Ronald

10

Ronald,

I am glad to hear there may be a way.
If you are using our OpenTracing bridge documentation, just make sure you go over the caveats section to make sure you are not using anything not supported.
We'll be happy to hear how it went and any feedback in general.

Good luck!

Eyal.

11

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.