OpenTelemetry Agent Integration - Kibana APM Dependency Operation Issue

Elastic Observability APM
,
1

I am migrating from Elastic APM Agent to OpenTelemetry Agent. The integration is working very well, exception one issue that is slowing us down when identifying the problem:
From Dependency, select Operation, then select one span, Latency and Throughtput dashboards are working, but the Trace dashboard said The selected trace cannot be found

span_ui
span_ui1907×921 102 KB

Any suggestion? Thank a lot everyone.

2

Hi,

This very probably comes from the data model difference between otel and our current data model.

  • Elastic APM has the concept of transactions, which are "root spans" when something is executed in a service (for example an http request).
  • with OTel everything is a span and there is no transaction, so the apm-server has to infer which are spans and which are transactions.

Here you have a transaction that has been created from a database span, and likely outside of any parent span (for example when the application starts), thus the trace might not be available.

Once the application has started and you have real user usage, do you still see such transactions without traces ? Does the issue is the same for all transactions or only specific to this one ?

3

Thanks for your response Mr. Sylvain,
This issue happens with all the dependencies, http, database, redis.
Yes, all my data is from real user usage (these span are created by real transactions and can be seen in the trace sample UI).
I have checked the data between elastic apm agent and otel agent, it seems that the data of spans from otel agent do not have trace.id field. This might be the root cause.

4

Sorry, I means there is no transaction.id field in the span data of otel agent.

5

These are images from 2 span:

  1. otel agent span

    otel_span
    otel_span1352×443 43 KB

  2. elastic apm agent span

    elasticapm_span
    elasticapm_span1351×473 37.6 KB

Can i try to set transaction.id equals parent,id in otel agent span?

6

Hi, I don't know why this is happening as the transaction.id should be set by the apm-server when OTLP (opentelemetry protocol) intake is used.

I'll ask the APM server team about this, which version of the stack (elasticsearch kibana apm-server) are you using here ?

7

Hi thanks for your reply,
Setting the transaction.id to the parent,id solves the issue " The selected trace cannot be found"

My stack runs version 8.11.4

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.