Elasticsearch Rest High Level Client is not traced

Elastic Observability APM
1

Kibana version: 6.8.0

Elasticsearch version: 6.8.0

APM Server version: 6.8.0

APM Agent language and version: JAVA apm-agent-attach:1.10.0, apm-agent-api:1.10.0

Browser version: Chrome 77.0.3865.90

Java version: openjdk:11

Description of the problem including expected versus actual behavior:
We are running a spring boot application (2.1.4.RELEASE) with rest api, which makes queries to Elasticsearch (7.3.0) using java high level client:

 <dependency>
        <groupId>org.elasticsearch.client</groupId>
        <artifactId>elasticsearch-rest-high-level-client</artifactId>
        <version>7.3.0</version>
    </dependency>

Though it is defined in the docs, that tracing of ES rest client comes out of the box, we cannot see any traces for it

We can see:

  • traces of the initial rest call
  • traces of the calls to downstream systems
  • traces of the @CaptureSpan annotated methods

Is elasticsearch-rest-high-level-client not supported by APM out of the box or we are missing smth.

we start APM upon spring application startup using:
ElasticApmAttacher.attach();

our config is:

active=true
service_name= {app-name}
server_urls= {apm-host}
transaction_sample_rate=0.5
application_packages=com.app
log_level=DEBUG

I something missing in our set up or there is no such support provided?

2

Hi Svetlana and welcome to the forum :slight_smile:

I couldn't reproduce this by adjusting our integration tests to use the 7.3 client (meaning- it is properly traced using the 7.1 integration test).

Please set log_level to DEBUG and upload the logs.

A few questions:

  1. Are these queries being performed synchronously as part of the request handling thread, or is there some async forking of some kind (eg usage of BlockingQueue) making the queried being done by another thread?
  2. Which APIs are you using? If you can send specific code snippet of a query not being traced, that would be useful.

Regardless, do you get your "initial rest calls" and "calls to downstream systems" collected as single or separate traces? If you are using supported technologies, then our distributed tracing mechanism should correlate them.

3

Hi Eyal_Koren,
Thank you for your help
It has turned out, that my colleauges shaded high level rest client jar to be able to connect to both ES5 and ES7 instances, which we have
since the package of ES7 was shaded, APM couldn't see it
but it worked out with the normal (non-shaded) jar

4

Glad to hear :slight_smile:.
Thanks for the feedback!