version: v8.17.3
setup
Deployed on elastic cloud
Services deployed on Kubernetes
Services instrumented with Opentelemetry Go Agent and kotel package (comes with frankz-go)
Services Communicate via Redpanda (Kafka compatible)
On Producing service side,
the producer is auto-instrumented using "github.com/twmb/franz-go/plugin/kotel"
kgo.WithHooks(kotelService.Hooks()...) where kotelService is simply *kotel.Kotel
This correctly adds the required headers on the produced records.
On consumer side,
consumer is also auto instrumented in the same way. In addition, after consuming a record I have ctx, span := KotelTracer.WithProcessSpan(record) where KotelTracer is simply *kotel.Tracer and record is a *kgo.Record. This ctx is used for rest of processing.
Description of the problem:
In transaction section on kibana,
I can see distributed tracing as expected with correct trace header propagation.
But on service Map I can see 2 different types of Connection.
- Service-A -> Redpanda Topic -> Service-B
- Service-A -> Service-B
expected Behaviour
Service-A -> Redpanda Topic -> Service-B
In the consumer service,
If I remove WithProcessSpan and create a normal span with otel.Tracer.Start then my distributed tracing stops working i.e., traces from service-B doesn't show up anymore in the traces from service-A. So, naturally Service-A -> Service-B direct connection disappears. The link between topic and consuming service stays because of the kafka consumer is auto instrumented. But the record headers aren't propagated to rest of the processes even if I use the same context.
And if I don't use kgo.WithHooks(kotelService.Hooks()...) as kgo options during kgo client creation, the link between the topic and the consuming service vanishes. And the receive span (i.e., the span that shows the receive process) doesn't show up on tracing.