# APM distributed tracing with Kafka **URL:** \ **Category:** APM\ **Tags:** java\ **Created:** [March 30, 2021, 9:30am UTC](https://discuss.elastic.co/t/apm-distributed-tracing-with-kafka/268759 "2021-03-30T09:30:14Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![shifenglim](https://avatars.discourse-cdn.com/v4/letter/s/b782af/32.png) [@shifenglim](https://discuss.elastic.co/u/shifenglim)\ **Post date:** [March 30, 2021, 9:30am UTC](https://discuss.elastic.co/t/apm-distributed-tracing-with-kafka/268759/1 "2021-03-30T09:30:15Z")
I am trying to implement APM for a system's infrastructure which consists of kafka and many java API services and most of the flows will have: api-1 =\> kafka =\> api-2 Right now, we enabled APM for api-1 and api-2 but there is a kafka in between and I am unsure 1. how we can do distributed tracing with kafka in the middle 2. are we able to monitor kafka using APM too? ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [March 30, 2021, 12:20pm UTC](https://discuss.elastic.co/t/apm-distributed-tracing-with-kafka/268759/2 "2021-03-30T12:20:53Z")
Hi and thanks for the question. > [@shifenglim](#): > > how we can do distributed tracing with kafka in the middle If you are using Kafka clients (and not Kafka streams) and a [supported Kafka version](https://www.elastic.co/guide/en/apm/agent/java/current/supported-technologies-details.html#supported-messaging-frameworks), than this should work. If that doesn't work, several questions: 1. What Java agent version are you using? 2. Do you get transactions for api-1? If not, then you would have to manually create one. Look in the [supported technologies page](https://www.elastic.co/guide/en/apm/agent/java/current/supported-technologies-details.html) to see what may produce transactions. 3. What Kafka broker version are you using? 4. What Kafka client version are you using? 5. If you set `log_level` to `DEBUG`, can you see some hints on what's traced and what's not? > [@shifenglim](#): > > are we able to monitor kafka using APM too? No, monitoring Kafka with Elastic would include [metrics](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-kafka.html#metricbeat-module-kafka) and [logs](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-kafka.html#filebeat-module-kafka), but not APM. ---
**Author:** ![shifenglim](https://avatars.discourse-cdn.com/v4/letter/s/b782af/32.png) [@shifenglim](https://discuss.elastic.co/u/shifenglim)\ **Post date:** [April 7, 2021, 4:05am UTC](https://discuss.elastic.co/t/apm-distributed-tracing-with-kafka/268759/3 "2021-04-07T04:05:40Z")
Hi, sorry for the late reply From what I heard from the team, they are using kafka client so from api-1 =\> kafka, the trace from api-1 captures the expected trace which also shows kafka. However the trace stops here and does not link to api-2 when the traceparent header is enabled we are using java agent 1.18 and to get transactions we are using the @CaptureTransaction annotation ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [April 7, 2021, 7:11am UTC](https://discuss.elastic.co/t/apm-distributed-tracing-with-kafka/268759/4 "2021-04-07T07:11:52Z")
Normally, you don't need to use `@CaptureTransaction` or anything else in order to get distributed tracing with Kafka. What happens if you remove any manual tracing from both services? > [@shifenglim](#): > > api-2 when the traceparent header is enabled Do you see Kafka records reaching api-2 service with the `traceparent` header? If the header is reaching api-2 and transactions don't get linked to the trace after removing any manual tracing, please provide `debug` log for api-2. If records are reaching api-2 without the header, please provide `debug` log for api-1. You may be able to understand what happens from the logs yourself. ---
**Author:** ![shifenglim](https://avatars.discourse-cdn.com/v4/letter/s/b782af/32.png) [@shifenglim](https://discuss.elastic.co/u/shifenglim)\ **Post date:** [April 7, 2021, 7:48am UTC](https://discuss.elastic.co/t/apm-distributed-tracing-with-kafka/268759/5 "2021-04-07T07:48:54Z")
at first our traces are not appearing with captureTransaction as we suspect most of the functions we wanted to trace are not http calls but internal local calls, adding captureTransaction to the local functions we want to trace will allow it to appear in kibana, As for the log\_level debug, where should I set this to begin debugging? Would it be in the APM level, or application level? Sorry I am new to debugging this tracing i will check on the api-2 part and get back to you ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [April 7, 2021, 9:52am UTC](https://discuss.elastic.co/t/apm-distributed-tracing-with-kafka/268759/6 "2021-04-07T09:52:15Z")
Try removing the `@CaptureTransaction` from anything that can be on the Kafka receiver part (i.e. api-2). > [@shifenglim](#): > > As for the log\_level debug, where should I set this to begin debugging? Would it be in the APM level, or application level? I was referring to the APM Java agent [`log_level`](https://www.elastic.co/guide/en/apm/agent/java/current/config-logging.html#config-log-level) config. Either through properties file or central config as `log_level`, or through a System property called `elastic.apm.log_level`, or an env variable called `ELASTIC_APM_LOG_LEVEL`, ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 28, 2021, 5:52am UTC](https://discuss.elastic.co/t/apm-distributed-tracing-with-kafka/268759/7 "2021-04-28T05:52:58Z")
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.