How to grab kafka traceparent and tracestate headers？

[wajika]

We use dapr framework, which includes opentelemetry trace, all trace data is sent to apmserver, and finally queried on kibana.
Everything is fine, but I noticed that if the request goes through kafka (or other middleware), then the whole request chain is broken, as shown in the picture below.

Is there any way to integrate kafka?

[GregKalapos]

Hey @wajika

the profiler based agent (which is currently in beta) has built in kafka support with auto instrumentation (in other words: you just add the agent and it'll be able to trace across kafka calls). Here is the docs of the profiler based agent. Have you tried that?

If that does not work, the other option is to do this manually with the public API. Here is how you can manually propagate tracing context - you can just call those methods and pass the string you get back via Kafka. If you do that, the whole chain will be in a single trace.

[wajika]

Thank you for your reply.
I believe I should be able to get kafka spans if I use APM agent directly.
service (apm agent) >> apmserver >> es
But let me clarify, we are currently using opentelemetry dotnet client (it is similar to apm agent), its data is sent to apmserver and then stored in Elasticsearch and finally displayed on kibana.
the path is like this
service (otel sdk) >> otlp-collector >> apmserver >> es
We've been using it for a while and didn't find the markup for kafka on the kibana apm UI.

The code will operate kafka at the red box mark, but it is not displayed on the UI.

I can't determine whether the otel client does not get the kafka header or the apmserver cannot integrate the kafka span. Is there any way to debug it?
BTW: I am using elastic stack 7.16.3

[GregKalapos]

I see,

well, you could turn on debug logs on APM Server to see how the OTel data looks which comes in - that'd be one options.

Or you can ask directly in the OTel GitHub repo: GitHub - open-telemetry/opentelemetry-dotnet: The OpenTelemetry .NET Client

[wajika]

Well, the .NET Client community isn't very happy to answer any questions.