OpenTelemetry span events are not visible in APM UI

[pskpr (Pushpak Prateek)]

Kibana version: 8.4.1

Elasticsearch version: 7.11.0

APM Server version: 8.4

OpenTelemetry version: 1.1.0

Browser version: Chrome 91

I am using a test version of elastic cloud. The index pattern and index templates are default as provided.

I am using OpenTelemetry to add tracing to a distributed NodeJs service. Further I use an otel collector hoisted on k8s to export the traces to ElasticAPM. The span events are visible in elastic index and there are supporting data streams as well. But these span events are not visible in APM UI as it should be as mentioned [here] ([OpenTelemetry] Support OpenTelemetry Span Events · Issue #4715 · elastic/apm-server · GitHub).

I use BasicTracerProvider from openTelemetry/sdk-trace-base to create a provider object.
The exporter is openTelemetry/exporter-trace-otlp-http to export the traces over http.
The url I use is /55681/v1/traces

There are no errors any where, neither in local console nor in k8s pod console.

Where should I check for errors to fix this issue?

Any help will be appreciated.

[axw (Andrew Wilkins)]

@pskpr

I am using a test version of elastic cloud

@pskpr can you please clarify this? Are you using Elastic Cloud? If so, I don't understand the versions you've specified above: you can't specify different versions for Elasticsearch, Kibana, and APM Server is Elastic Cloud.

The span events are visible in elastic index and there are supporting data streams as well. But these span events are not visible in APM UI as it should be

If you can see them in the logs-apm.app-default data stream, then I suppose it's a UI issue. What do you see in the UI? Perhaps you are running into this bug? Logs cannot be displayed in APM Traces after upgrade to 8.4.0 · Issue #139631 · elastic/kibana · GitHub

[pskpr (Pushpak Prateek)]

Sorry I'm quite new with this. I see this is the same bug that you have pointed out. Waiting for it to be fixed.
I am getting this in the logs

[axw (Andrew Wilkins)]

No worries. Unfortunately you are running into a known bug in Kibana. This will be fixed in 8.4.2, which should be out in the coming days.

[pskpr (Pushpak Prateek)]

Thanks for the guidance @axw . Cheers

[pskpr (Pushpak Prateek)]

Hi @axw, I rolled back to version 8.3.2 of kibana and now i'm able to get the span events. The issue is i can only see the name of the event
for instance

span.addEvent('get data from elastic', {
event: 'Success',
message: 'Get data from elastic Success'
})

i only see

though that data is visible in elastic index as labels,

it is not visible in APM UI

[axw (Andrew Wilkins)]

@pskpr you're right, these additional details are not currently visible in the Logs component - it's not ideal.

Please take a look at [APM - OpenTelemetry] Span Events and Logs support: enhancement requests · Issue #123256 · elastic/kibana · GitHub, and let us know what you think of the enhancements described there. If you think they sound good, I'd suggest giving it a and subscribing to find out when it is implemented.

[pskpr (Pushpak Prateek)]

Thanks a lot for the reply @axw . And thanks for the help, i'll look into it figure the rest. Cheers