Span details not visible in APM traces waterfall

Elastic Observability APM
1

Dear All,

I am manually instrumenting my apps using open telemetry. Also adding logs into trace attributes. Kind of integrating logs with traces.
I send traces to APM server. When I am at trace waterfall view I see only transaction details. when I click on a span I don't see any span details.
Could the differences in how traces exist in elastic have an impact. I have added traces of a transaction and a span at the bottom.
Please help me find out why.

image
image774×746 59.2 KB

image
image848×771 33.5 KB

This is how my traces exist in APM index.
Transaction trace:

          "agent": {
            "name": "otlp",
            "version": "unknown"
          },
          "processor": {
            "event": "transaction"
          },
          "url": {
            "scheme": "http",
            "full": "http:"
          },
          "labels": {
            "timeStamp": "2025-02-11T11:37:22Z",
            "http_route": "/check_api",
            "custom_uuid": "000",
            "custom_response_message": "To check"
          },
          "observer": {
            "hostname": "9181bc81124e",
            "type": "apm-server",
            "version": "8.17.0"
          },
          "trace": {
            "id": "42831d2c951af72675397d721aa6bea8"
          },
          "@timestamp": "2025-04-01T08:57:08.511Z",
          "data_stream": {
            "namespace": "default",
            "type": "traces",
            "dataset": "apm"
          },
          "numeric_labels": {
            "custom_error_code": 50001,
            "http_response_duration": 230000000
          },
          "service": {
            "framework": {
              "name": "springboot-app"
            },
            "name": "AAA",
            "language": {
              "name": "unknown"
            }
          },
          "http": {
            "request": {
              "method": "POST"
            },
            "response": {
              "status_code": 200
            }
          },
          "event": {
            "success_count": 1,
            "outcome": "success"
          },
          "transaction": {
            "result": "HTTP 2xx",
            "duration": {
              "us": 1125753
            },
            "representative_count": 1,
            "name": "root-span",
            "id": "47816b211f9ed8e3",
            "type": "request",
            "sampled": true
          },
          "span": {
            "id": "47816b211f9ed8e3"
          },
          "timestamp": {
            "us": 1743497828511800
          }
        }
      }

Span trace:

        "_source": {
          "observer": {
            "hostname": "9181bc81124e",
            "type": "apm-server",
            "version": "8.17.0"
          },
          "parent": {
            "id": "47816b211f9ed8e3"
          },
          "agent": {
            "name": "otlp",
            "version": "unknown"
          },
          "trace": {
            "id": "42831d2c951af72675397d721aa6bea8"
          },
          "@timestamp": "2025-04-01T08:57:08.516Z",
          "data_stream": {
            "namespace": "default",
            "type": "traces",
            "dataset": "apm"
          },
          "service": {
            "framework": {
              "name": "springboot-app"
            },
            "name": "AAA",
            "language": {
              "name": "unknown"
            }
          },
          "event": {
            "success_count": 1,
            "outcome": "success"
          },
          "processor": {
            "event": "span"
          },
          "span": {
            "duration": {
              "us": 609559
            },
            "representative_count": 1,
            "subtype": "internal",
            "name": "greeting-api",
            "id": "57e3866f3b44f7c7",
            "type": "app"
          },
          "timestamp": {
            "us": 1743497828516292
          }
        }
      }
2

Hi,

You are using the opentelemetry SDK to manually capture those traces and send them to the APM Server OTLP endpoint.

What is probably happening here is that some required attributes for the UI to work properly are missing and thus you can't properly display the spans in the trace view.

Maybe to identify what is missing the easiest way would be to add the otel instrumentation agent or our Java EDOT agent to your application, this should allow to send all the required data.

Doing this should make the instrumented application send data that should properly be usable in the UI, once you have that it would be a matter to identify what is missing and then find why it's missing, for example maybe there is something off in your otel SDK setup.

3

Thank you @Sylvain_Juge for your response. I observed the traces. There are no transaction related fields in child span traces. APM server isn't mapping these fields. This could be a reason for missing span details. Yet to figure out how to get traces in right format.

Regards
Meghana