Java APM agent does override default method, causing no-op tracing

Elastic Observability APM
,
1

Elasticsearch Java client version
8.11.1

Kibana version:
8.9.1

APM Elasticsearch version:
8.9.1

APM Server version:
8.9.1

APM Agent language and version:
Java, 1.46.0

Hello,
I originally wanted to create github issue, but your guidelines recommend creating post here first, so here we go:

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

Java Elastic APM agent does not record OpenTelemetry spans for Elasticsearch client.

Prelude
Java Elasticsearch client >= v8.10 comes with its own instrumentation. Here I am talking specifically about ElasticsearchTransportBase.

When no instrumentation is specified via constructor, then OpenTelemetryForElasticsearch.getDefault() is called, which in turn calls GlobalOpenTelemetry.get(), which Java Elastic APM agent hooks and makes the method return OpenTelemetry object of implementation co.elastic.apm.agent.opentelemetry.global.ElasticOpenTelemetryWithMetrics.
Finally, the the OpenTelemetry instance is passed into OpenTelemetryForElasticsearch constructor, which attempts to create Tracer object via openTelemetry.tracerBuilder("elasticsearch-api")...build() call. Therein lies a bug.

The bug - Actual behaviour
The TracerBuilder, that is provided by calling getTracerProvider().tracerBuilder(instrumentationScopeName) in the OpenTelemetry#tracerBuilder, is instance of DefaultTracerBuilder, because the co.elastic.apm.agent.opentelemetry.tracing.OTelTracerProvider does not override method - io.opentelemetry.api.trace.TracerProvider#tracerBuilder - which by default returns no-op instance. This I think causes the problem of Elasticsearch client spans not being recorded, since the OpenTelemetryForElasticsearch#tracer ends up being no-op.

Here is a picture of the resulting types from mentioned methods.

image
image1552×187 38.7 KB

Expected behaviour
ElasticOpenTelemetryWithMetrics#tracerBuilder(String) should not produce no-op builder.

Thank you

2

For now, I have made a workaround, where I put instance Tracer of type OTelTracer into the OpenTelemetryForElasticsearch#tracer field, that is being used in the ElasticsearchTransportBase instance, via reflection.

The tracer is created as follows:

    private Tracer getTracer() {
        OpenTelemetry openTelemetry = GlobalOpenTelemetry.get();
        Version version = Version.VERSION;

        return openTelemetry.getTracer("elasticsearch-api", version == null ? "unknown" : version.toString());
    }

After this workaround, spans created by the Elasticsearch Java client are correctly correlated to the current APM transaction and arrive to the APM server

3

Thanks for reporting this, the team will look into it as soon as we can!

4

Thank you :+1:

5

Thanks again for the detailed report @bebeDobreRano , you can follow this issue to get notified when this is fixed.

6

Will follow. Thank you for the fix :+1:

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.