OpenTelemetry traces are not properly shown on Kibana when using otel dotnet agent

Elastic Observability
1

Hey, i having a small issue when trying to visualize traces in kibana apm, each are sent by opentelemetry agent. The issue is on the detail the traces appear on the timeline. When i using elastic apm agent i have better information : "GET dummy-microservice:8082", for my example. And i dont need to do a extra click to see each service the call goes to. Please, see image below.

When sending trace with otel agent :

image
image1703×195 10.7 KB

Seems that elastic apm server uses "span.name" as label in timeline. I was able to fix the issue ( even thought it looks ugly ) when using opentelemetry as code :

otel.WithTracing(tracing =>
{
    tracing.AddAspNetCoreInstrumentation();
    tracing.AddHttpClientInstrumentation(options =>
    {
        options.EnrichWithHttpRequestMessage = (activity, httpRequestMessage) =>
        {
            // Extract domain and port from the request URI
            var domain = httpRequestMessage.RequestUri.Host;
            var port = httpRequestMessage.RequestUri.Port != 80 && httpRequestMessage.RequestUri.Port != 443
            ? ":" + httpRequestMessage.RequestUri.Port.ToString()
                       : string.Empty;

            var method = httpRequestMessage.Method.Method;  // Get HTTP method (e.g., GET)

            // Set custom span name
            activity.DisplayName = $"{method} {domain}{port}";
        };
    });

However when using otel with autoinstrumentation setup, this is not possible to do.

Using :

  • elasticsearch | kibana | apm-server -- all version 8.15.0
  • otel dotnet libs -- 1.9.0
  • otel dotnet autoinstrumentation -- 1.7.0

Is there is any way to get around this issue in other "pretty" way ? Could the elastic team consider this issue ?

Thank you in advance

2

Hey, @JoaoCampos07.

Unfortunately, there will need to be some manual intervention if you prefer a different span name (which, as you mentioned, is used to label the span in the trace). The .NET OpenTelemetry SDK follows the semantic conventions for HTTP client span naming, and right now, those require the span name to include only the HTTP method. There is an experimental attribute which may, in the future, lead to a URL template being used when available in the instrumentation, but this would first need to be made stable and then require changes in the OTel instrumentation and potentially even the .NET library itself, so won't change for the foreseeable timeframe.

Your approach for NuGet-based installation is likely the best solution you have right now. Another option is defining and registering a Processor. If you want an option that can also support auto-instrumentation, you would need to look at the experimental plugin functionality of the profiler-based auto-instrumentation. That can be used to configure the Tracer and register additional functionality before it's built.

The Elastic APM agent takes a different stance on naming the span, which includes the host and port. I'm unsure if OTel ever considered that, as it would be low cardinality.

1 Like
3

Further to the above, the discussion with the OTel semantic convention WG was met with some explanations for the low-cardinality design, but they are open to me raising and issue to discuss if at least appending the host as the fallback for the target would be considered.

Internally, we are discussing this as well. There's potential for us to consider enhancements in Kibana to add this but that needs some time to review the options. We might also consider adding this to our distro.

A final thing to add is that you could also consider adding a collector into your pipeline and modify the span name there. That would require no per-service changes as you could centralise the behaviour. That's one of the benefits of shipping via a collector vs. sending directly to a backend.

4

Thank you very much, Steve. For the tips to solve it and the general view.

For us and for now, we can live with the extra click to check each microservice was called. Maybe we will register the processor later on, as we really want to have this auto-instrumented (maybe meanwhile there will be changes in APM to adapt to that :stuck_out_tongue: ).

Just, one final question about this part : " but they are open to me raising and issue to discuss if at least appending the host as the fallback for the target would be considered" . So there is github issue open ? Just would like to track this discussion if is public.

5

You're welcome. Here is the issue I opened today.