Missing child spans on OpenTelemetry traces

[Sebastien_Dan]

Kibana version: 8.9.0

Elasticsearch version: 8.9.0

Original install method (e.g. download page, yum, deb, from source, etc.) and version: Elastic Cloud, no APM agent

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

Child spans are not displayed on Kibana.

My setup is the following: my instrumented web app sends OpenTelemetry traces to my otel collector which exports them to Elastic Cloud.

Printing extra logs from the otel collector, I rightfully get the following:

2023-08-08T17:09:54.620Z	info	TracesExporter	{"kind": "exporter", "data_type": "traces", "name": "logging", "#spans": 2}
2023-08-08T17:09:54.620Z	info	ResourceSpans #0
Resource SchemaURL:
Resource attributes:
     -> service.name: Str(dapp-frontend-erc20-messaging)
     -> telemetry.sdk.language: Str(webjs)
     -> telemetry.sdk.name: Str(opentelemetry)
     -> telemetry.sdk.version: Str(1.15.1)
     -> service.version: Str(0.1.0)
ScopeSpans #0
ScopeSpans SchemaURL:
InstrumentationScope dapp-frontend-erc20-messaging 0.1.0
Span #0
    Trace ID       : 5633a95f60cc533d9432636b995ba427
    Parent ID      : f406221c88072f31
    ID             : 550fce0f887494d2
    Name           : step-0
    Kind           : Internal
    Start time     : 2023-08-08 17:09:48.566000128 +0000 UTC
    End time       : 2023-08-08 17:09:51.333199872 +0000 UTC
    Status code    : Unset
    Status message :
Span #1
    Trace ID       : 5633a95f60cc533d9432636b995ba427
    Parent ID      :
    ID             : f406221c88072f31
    Name           : multi-step-form
    Kind           : Internal
    Start time     : 2023-08-08 17:09:48.564999936 +0000 UTC
    End time       : 2023-08-08 17:09:51.333299968 +0000 UTC
    Status code    : Unset
    Status message :
	{"kind": "exporter", "data_type": "traces", "name": "logging"}

I.e., a simple multi-step-form root span that contains one single step-0 child span. From these logs above I understand my instrumentation is working as expected.

The problem is that on Kibana/APM, except a few rare and seemingly random times, I only see the root span:

Even more surprisingly, if my child span emits events, I do see them as "Logs" of the root span.

If anyone has any idea what's going on, I'm all ears!

[alexander.wert]

Hi @Sebastien_Dan ,

thanks for reporting this!

When searching in Kibana's Discover for the corresponding trace id within the APM data view (with the following query in the query bar), do you see both spans in the results or only the parent span?

trace.id: "5633a95f60cc533d9432636b995ba427"

[Sebastien_Dan]

Hey @alexander.wert, thanks for replying! (which just happened coincidentally when I looked at the page!)

When searching on Discover, I do not find the missing spans (note: in the very rare occasion when the APM view did show one of the child spans (but one only), it was also findable on Discover).

[Sebastien_Dan]

Hey @alexander.wert, would you have any lead on this?

[Sebastien_Dan]

@alexander.wert FYI, I tested another approach with local elastic APM agents sending data directly to Elastic (from both the browser and my nodejs server) instead of otel lib -> otel collector -> Elastic, and I do not get the issue. So perhaps there's a otel collector -> Elastic parsing issue here?