Elastic Java Client Async APM Transaction

[elastic_rookie (Rookie)]

If you are asking about a problem you are experiencing, please use the following template, as it will help us help you. If you have a different problem, please delete all of this text

TIP 1: select at least one tag that further categorizes your topic. For example server for APM Server related questions, java for questions regarding the Elastic APM Java agent, or ui for questions about the APM App within Kibana.

TIP 2: Check out the troubleshooting guide first. Not only will it help you to resolve common problems faster but it also explains in more detail which information we need before we can properly help you.

Kibana version: 8.53

Elasticsearch version: 8.5.3

APM Server version:8.5.3

APM Agent language and version: Java / 8.5.3

Browser version:Chrome 108

Original install method (e.g. download page, yum, deb, from source, etc.) and version:

Fresh install or upgraded from other version? Fresh

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.
Nothing special, basic docker-compose up setup.

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

I'm currently trying to instrument a transaction through an entire API service call, but seem to lose the ElasticApm.currentTransaction/Span information after any async call to Elasticsearch.

I setup an extremely simple test application with Jersey endpoints that receive a request. The Java agent automatically instruments the Jersey request and any method I call that has the appropriate Public Api annotations on them. This works for any synchronous calls and asynchronous methos (ExecutorService / CompletableFutures), except when I perform an async request with the Java Elastic client. Why is the context not propagating to any future stages after the call to elastic?

Here's a basic structure of the call I'm making

ExecutorService _executorService = Executors.newFixedThreadPool(10);

return _asyncClient.search(s -> s
                        .index("products")
                        .query(q -> q
                                .term(t -> t
                                        .field("name")
                                        .value(v -> v.stringValue("bicycle"))
                                )),
                Product.class)
                .thenApplyAsync(response -> {

   /* THE BELOW VALUES ARE ALWAYS NOOP INSTANCES */
                    Transaction trans1 = ElasticApm.currentTransaction();
                    Span span1 = ElasticApm.currentSpan();
                    LOGGER.info("[Apm trans/span id -- {}/{}", trans1.getId(), span1.getId());

                    return response;
                }, _executorService);

Any subsequent stage doesn't have the Trace information and consequently the Public Api annotations don't pick anything up either.

Am I misunderstanding how this should be instrumented with Java Futures and the instrumentation from Executors in APM?

Thanks in advance!

[Sylvain_Juge]

Hi @elastic_rookie , thanks for reporting this !

I think it requires further investigation, but I have managed to reproduce a similar case as you reported, and I get the same result, thus it is probably a bug in the agent or something in the ES client that prevents the agent to work as expected.

Thus here the next step is on us to further investigate and fix the issue as this is not the expected behavior.

[elastic_rookie (Rookie)]

Hi @Sylvain_Juge ,

Thanks for investigating and letting me know! Do you have a ticket/issue link I can track to know when it's fixed?

Are there any potential workarounds that can be applied in the interim?

Thanks!

[Sylvain_Juge]

Hi @elastic_rookie ,

I haven't forgotten about your issue, and I am currently trying a few approaches how to fix this.
There isn't any issue or PR because I still need to experiment a bit with a few different approaches.

I'll keep you posted on the progress, hopefully there should be a snapshot version to test soon , would you be willing to try it on your application to make sure it works for you ?

[elastic_rookie (Rookie)]

@Sylvain_Juge sure thing, just let me know when it's ready to test!

[Sylvain_Juge]

Hi !

I have just opened a PR with a fix that should provide the active transaction/span in the CompletableFuture that you get from the 8.x Elasticsearch client.

Could you test with the following snapshot to confirm your use-case is properly covered ?

[elastic_rookie (Rookie)]

@Sylvain_Juge This fixed work great for the 8.x client! In my simple test setup the transaction/span ids were picked up appropriately on subsequent future chains.

Do you know if this change will also work for the 7.X clients as well?

Looking forward to having this fixed, thanks for the support!

[Sylvain_Juge]

Thanks for double-checking this @elastic_rookie !

This fix actually is implemented in the 7.x REST client (the 8.x client also relies on REST client), so the recent versions of 7.x Elasticsearch clients are covered as well. In the added unit tests you have an example of an equivalent code for 7.x.

Thanks again for the feedback, this will likely be shipped in the next agent release.
You can subscribe to release notifications on github to get notified when it's available.