Dependencies don't show up in APM

[Willie_Abrams]

I have an app running in Cloud Run that I am profiling with Elastic APM. I get transaction level data, all the normal APM stuff you see on the Overview page. Transactions are populated. I get Trace samples. I click on a trace sample and I see my [db][Elasticsearch] spans. I click on an Elasticsearch span and the backend is "Elasticsearch" -- everything is right, even the "View span in Discover" link that takes me to the relevant data in Kibana. All of it is great, EXCEPT 2 problems:

1. If I click on the Backend: Elasticsearch link, it goes to a Dependencies>Elasticsearch page with no data. No latency graph, no throughput, nothing.

2. If I click on the Service Map, I see my Elasticsearch server there. If I click on it, I get "No data for selected environment. Try switching to another environment."

It's baffling and I've been going round and round with support on it. Here are my details:

APM agent: go.elastic.co/apm v1.14.0
Kibana, Elasticsearch: 7.15.0 (this is an elastic.co managed instance in GCP)

FWIW, I do have New Relic agent installed as well, but I have a very hard time removing it from a production instance as it provides alerting, etc. Part of the reason I am trying to see if Elastic APM is an alternative.

[Willie_Abrams]

FWIW, I've removed the New Relic agent and I'm still having the same problem.

[stephenb (Stephen Brown)]

Apologies I'm a little confused, you're looking at using elastic APM And you're also using Elasticsearch as a datastore as part of the go microservice?

Which is fine Just trying to understand the landscape.

Are you generating transactions That would access Elasticsearch data store?

How long did you let it run? How many transactions?

Can you show a picture of your service map?

Can you show a picture of the dependent services page.

Also we released recently 7.15.1 I think there may have been a few bugs fixes perhaps upgrade should be a one click upgrade on Elastic Cloud.

[axw (Andrew Wilkins)]

@Willie_Abrams welcome to the forum!

I just set up a basic Cloud Run application instrumented with the Elastic APM Go agent, and I have found that Cloud Run is adding its own Trace-Context header to the request headers:

Because of this, Elastic APM does not know the sampling ratio (if < 100%) applied, and so it does not produce the required metrics that power the Dependencies list.

I'll bring this up with the rest of the team and see what we can do about it.

In the short term, one thing you could do is wrap your instrumented handler such that the traceparent header is removed. Something like this:

mux := http.NewServeMux()                                                    
mux.HandleFunc("/", handler)                                                 
wrapped := apmhttp.Wrap(mux)                                                 
var handler http.HandlerFunc = func(w http.ResponseWriter, r *http.Request) {
        r.Header.Del("Traceparent")                                          
        wrapped.ServeHTTP(w, r)                                              
}                                                                            

[Willie_Abrams]

This worked! Thank you. Obviously I'd love to not have to delete it, as I'm unaware of what ramifications this has on other observability aspects, but this definitely worked. Because of this, I found Traceparent header from outside Elastic · Issue #286 · elastic/apm · GitHub and I'll subscribe to that issue. Are there other issues I should follow?

[axw (Andrew Wilkins)]

Good find, #286 is exactly what I was intending to link after coming back from discussing with the team I'm not aware of any other issues that are relevant.

Ideally you wouldn't need to manually configure anything for instrumented apps to work in Cloud Run, so we may amend #286 to identify that and automatically restart the trace by default.