Missing outgoing HTTP Span GO Lang

[ianmcgraw]

Kibana version: v 7.9.2 (elastic cloud)

Elasticsearch version: v 7.9.2 (elastic cloud)

APM Server version: v 7.9.2 (elastic cloud)

APM Agent language and version: Golang

	go.elastic.co/apm/module/apmgorilla v1.10.1-0.20210330103029-f759010eb3b4
	go.elastic.co/apm/module/apmhttp v1.11.0

Hi,

I recently instrumented our Golang services with the ElasticAPM go agent. So for I have setup multiple services that successfully trace outgoing http requests to downstream services. Unfortunately, our arthurai-metric-service seems to be missing traces in the parent service's transactions. Here is a screenshot of the parent trace:

There should be another span for an outgoing http call before the POST call shown to arthurai-v3-schema-service in this trace. Oddly, the downstream service registers the trace and even has a link to this parent trace, but for some reason, it doesn't show in this parent trace. Here is a screenshot of the child trace:

If I click "view full trace" in this child trace, it brings me to the correct parent trace above, but for some reason this one is not linked like the other services are.

Additionally, If I look at the metadata of the downstream service's trace:

I see the same trace.id that I see in the other downstream call to arthurai-schema-service's trace:

It seems like the trace context is correctly passing to the downstream service. Is there something I'm missing here that should link these together? Thanks for the help!

[axw (Andrew Wilkins)]

Welcome to the forum @ianmcgraw!

One possible explanation is that the response body isn't being closed or fully consumed in the client -- either of these is what marks the end of the client HTTP span. Can you please check that first?

If it's not that, then we'll need to grab some debug logs for the agent: Troubleshooting | APM Go Agent Reference [1.x] | Elastic to see if the server is rejecting any events.

Finally, it would also be worth taking a look at the documents in Discover to rule out any bugs in the APM UI, unlikely as that may be. Filter on trace.id and see if there's a document with span.name set to POST arthurai-v3-metric-service.

[ianmcgraw]

Ah yes! We were missing the call to Body.Close(). No wonder I couldn't find where the span ended. Thanks so much!