APM distributed tracing with Kafka

Elastic Observability APM
1

I am trying to implement APM for a system's infrastructure which consists of kafka and many java API services and most of the flows will have:

api-1 => kafka => api-2

Right now, we enabled APM for api-1 and api-2 but there is a kafka in between and I am unsure

  1. how we can do distributed tracing with kafka in the middle
  2. are we able to monitor kafka using APM too?
2

Hi and thanks for the question.

If you are using Kafka clients (and not Kafka streams) and a supported Kafka version, than this should work. If that doesn't work, several questions:

  1. What Java agent version are you using?
  2. Do you get transactions for api-1? If not, then you would have to manually create one. Look in the supported technologies page to see what may produce transactions.
  3. What Kafka broker version are you using?
  4. What Kafka client version are you using?
  5. If you set log_level to DEBUG, can you see some hints on what's traced and what's not?

No, monitoring Kafka with Elastic would include metrics and logs, but not APM.

3

Hi, sorry for the late reply

From what I heard from the team, they are using kafka client

so from api-1 => kafka,

the trace from api-1 captures the expected trace which also shows kafka. However the trace stops here and does not link to api-2 when the traceparent header is enabled

we are using java agent 1.18 and to get transactions we are using the @CaptureTransaction annotation

4

Normally, you don't need to use @CaptureTransaction or anything else in order to get distributed tracing with Kafka. What happens if you remove any manual tracing from both services?

Do you see Kafka records reaching api-2 service with the traceparent header?
If the header is reaching api-2 and transactions don't get linked to the trace after removing any manual tracing, please provide debug log for api-2.
If records are reaching api-2 without the header, please provide debug log for api-1.
You may be able to understand what happens from the logs yourself.

5

at first our traces are not appearing with captureTransaction as we suspect most of the functions we wanted to trace are not http calls but internal local calls, adding captureTransaction to the local functions we want to trace will allow it to appear in kibana,

As for the log_level debug, where should I set this to begin debugging? Would it be in the APM level, or application level? Sorry I am new to debugging this tracing

i will check on the api-2 part and get back to you

6

Try removing the @CaptureTransaction from anything that can be on the Kafka receiver part (i.e. api-2).

I was referring to the APM Java agent log_level config. Either through properties file or central config as log_level, or through a System property called elastic.apm.log_level, or an env variable called ELASTIC_APM_LOG_LEVEL,

7

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.