[Following up on closed topic: https://discuss.elastic.co/t/apm-http-context-transferred-to-grpc-context/231421]
My backend is in golang and I'm using regular opentelemetry tooling: http://go.opentelemetry.io/otel (v1.34.0).
I start the trace as HTTP, then transcode the request and handle it as gRPC (in the same lifecycle, no remote proxy or anything).
// HTTP middleware
// Retrieve the span created by otelhttp.Handler
wrapper := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
span := trace.SpanFromContext(ctx)
ctx = context.WithValue(ctx, OtelRootSpanContextKey{}, span)
r = r.WithContext(ctx)
h.ServeHTTP(w, r)
})
// init otelhttp handler only once
handler := otelhttp.NewHandler(wrapper, "",
otelhttp.WithSpanNameFormatter(func(_ string, r *http.Request) string {
return fmt.Sprintf("%s %s", r.Method, r.RequestURI)
}),
otelhttp.WithSpanOptions(
trace.WithSpanKind(trace.SpanKindServer),
),
otelhttp.WithPropagators(prop),
otelhttp.WithTracerProvider(tp))
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
ctx := context.WithValue(r.Context(), OtelTracerNameKey, serviceName)
prop.Inject(ctx, propagation.HeaderCarrier(r.Header))
handler.ServeHTTP(w, r.WithContext(ctx))
})
gRPC part is set up using the new way via stats handler instead of an interceptor:
// gRPC Server
opts = append(opts, grpc.StatsHandler(otelgrpc.NewServerHandler(
otelgrpc.WithTracerProvider(tracer),
otelgrpc.WithPropagators(prop),
)))
server := grpc.NewServer(opts...)
// ... transcoder setup that wraps the grpc server and gives a http _grpcHandler_
// Propagate parent trace from HTTP into gRPC
otelWrapper = http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
tp := trace.SpanFromContext(r.Context()).TracerProvider()
ctx, span := tp.Tracer(OtelTracerNameKey,
trace.WithSchemaURL(semconv.SchemaURL),
trace.WithInstrumentationVersion(otelhttp.Version())).
Start(r.Context(), spanName, trace.WithSpanKind(trace.SpanKindServer))
defer span.End()
// inject
p.prop.Inject(ctx, propagation.HeaderCarrier(r.Header))
ctx = p.prop.Extract(ctx, propagation.HeaderCarrier(r.Header))
grpcHandler.ServeHTTP(w, r.WithContext(ctx))
// ... What am I missing?
})
// ... net/http serve
I had to properly massage the request by injecting and re-extracting the carrier into the context to make it all work and as shown in Screenshot 1, spans are correctly recognized as part of the same request and they have the same trace.id attribute.
However, as you can see from Screenshot 2, http and grpc parts are displayed as separate unique transactions in the service transactions list.
I wonder what else should I do for them to not be shown separately. This affects the stats and incorrectly shows 2x of the actual throughput.
I'm obsessed with Elastic APM and want to improve my monitoring setup.
Kibana version: 8.17.0
Elasticsearch version: 8.17.0
APM Server version: 8.17.0
APM Agent language and version: via opentelemetry
Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
Screenshot 1:
