OpenTelemetry Agent Integration - Kibana APM Dependency Operation Issue

[baoletrg (Bao Le Trong)]

I am migrating from Elastic APM Agent to OpenTelemetry Agent. The integration is working very well, exception one issue that is slowing us down when identifying the problem:
From Dependency, select Operation, then select one span, Latency and Throughtput dashboards are working, but the Trace dashboard said The selected trace cannot be found

Any suggestion? Thank a lot everyone.

[Sylvain_Juge]

Hi,

This very probably comes from the data model difference between otel and our current data model.

• Elastic APM has the concept of transactions, which are "root spans" when something is executed in a service (for example an http request).
• with OTel everything is a span and there is no transaction, so the apm-server has to infer which are spans and which are transactions.

Here you have a transaction that has been created from a database span, and likely outside of any parent span (for example when the application starts), thus the trace might not be available.

Once the application has started and you have real user usage, do you still see such transactions without traces ? Does the issue is the same for all transactions or only specific to this one ?

[baoletrg (Bao Le Trong)]

Thanks for your response Mr. Sylvain,
This issue happens with all the dependencies, http, database, redis.
Yes, all my data is from real user usage (these span are created by real transactions and can be seen in the trace sample UI).
I have checked the data between elastic apm agent and otel agent, it seems that the data of spans from otel agent do not have trace.id field. This might be the root cause.

[baoletrg (Bao Le Trong)]

Sorry, I means there is no transaction.id field in the span data of otel agent.

[baoletrg (Bao Le Trong)]

These are images from 2 span:

1. otel agent span
   

   

2. elastic apm agent span
   

   

Can i try to set transaction.id equals parent,id in otel agent span?

[Sylvain_Juge]

Hi, I don't know why this is happening as the transaction.id should be set by the apm-server when OTLP (opentelemetry protocol) intake is used.

I'll ask the APM server team about this, which version of the stack (elasticsearch kibana apm-server) are you using here ?

[baoletrg (Bao Le Trong)]

Hi thanks for your reply,
Setting the transaction.id to the parent,id solves the issue " The selected trace cannot be found"

My stack runs version 8.11.4