Kafka / Spring Boot 3.4 & Java APM Agent

[VVK (VVetsa)]

Self managed in K8S environment.
Kibana version:
8.16.5
Elasticsearch version:
8.16.5
APM Server version:
8.16.2
APM Agent language and version:
Java 1.52.1
Browser version:
Chrome

Fresh install or upgraded from other version?
Upgraded Version

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs?
Yes for Logstash & kafka outputs (for log processing); self managed Strimzi Kafka

Environment is: K8S
Application:
Java Dependencies:
Java : 17.0.6
Spring boot 3.4
Kafka spring version: 3.2.5
kafka version: 3.7.1

This is Doubt:
Kafka - Auto Instrumentation does it work for spring-kafka? Or is it that manual instrumentation has to be done? If so any recommendations to not to lose end to end traces?

Setup of Agent:

• As per documentation, tracing is enabled by using trace_methods.
• ECS logs enabled.
• Nothing has been defined for enable_instrumentations.

What does my application do?
Programmatic Self attach; ElasticApmAttacher.attach()
My Application acts as consumer ( I use @KafkaListener). It connects to grpc server, fetches data and posts to kafka a topic.

What is being aimed?
Aiming for end to end tracing (via rum).

Current state:
My listener application captures grpc/ http but not Kafka Producer (via Spring Kafka)

Found very old conversation where in there is recommendation of how things can be instrumented for spring kafka:

Spring-kafka instrumentation - KafkaListenerEndpointContainer mono-thread

Hi there, I currently have a micro-services architecture composed of Spring-boot micro-services and communicating synchronously via Rest and asynchronously via Kafka messages. While the synchronous communication is fully instrumented out of the box by the Elastic APM Java agent, Kafka on the other side isn't instrumented yet, so I'm trying to implement myself using the Public API. CODE DETAILS Producer The producer injects trace headers into the kafka message using a ProducerInterceptor K…

Subquestion: Is this above discussion still relevant today (depending on suitability) for manual instrumentation.

Application flow:
RUM enabled Client (1) -> Http Rest end point (2) -> grpc (3)-> kafka- producer (4) -> kafka-consumer(5) -> mongodb/ grpc/http / kafka producer(6).

Aiming for tracing: step 5 & step 6 (along with logs).

[Sylvain_Juge]

Hi !

The related discussion is from ~6 years ago, so it's probably not very relevant anymore.

For your use-case, I don't get where the issue is, can you elaborate a bit ?

In particular, you need to explain what is running where and the communication between the JVMs in-between. Also, can you also elaborate on where you have to use the agent API and trace_methods configuration ?

For example, does the http rest endpoint is making a grpc client call to another grpc backend which then calls a kafka producer or is that within the same JVM ?

Also, the kafka service itself should not have to be instrumented, the messages sent to it should contain all the correlation IDs when they have been sent from an instrumented application.

[VVK (VVetsa)]

@Sylvain_Juge
Thank you for responding.

I am giving my rough pictorial diagram of of pods/ application jvms are there.

Let's concentrate on pod-3. There are three operations/calls: make a call to grpc server(pod4), make a call to MongoDB and publish a record in kafka to be consumed by pod5.

For pod-3, instrumentation is achieved is via trace_methods for method MyConsumer#processMessage.

Behaviour:
GRPC calls are captured by trace (blue).
But calls to MongoDB/kafkaTemplate are not captured. And also not listed in dependencies.
Short term target is also to capture both of these operation2 & operation3

Long term Target is to even capture how pod5 reacts (or as many consumergroups). This will get an entire end to end trace. For this I was referring to possible discussion that was carried out approximately six years back.

Below Rough pictorial diagram listed here might clarify some of confusions that my original post might have given and what I want to acheive.

[Sylvain_Juge]

I think it could be worth trying the new EDOT Java agent instead to see what is being captured on jvm/pod-3.

Are all the 3 operations executed synchronously when processing a message ?

There are a few differences where it could help:

• the need for trace_methods should be removed as all the spans are being captured with and without a transaction (unlike the Elastic APM Java agent which requires a transaction).
• it should have a more exhaustive coverage for instrumentation, which probably includes your use-cases with Kafka

When testing I would suggest to check those things in order:

1. do you get a span/transaction for every message received by your custom consumer ?
2. are the calls to mongodb and grpc properly captured ?
3. are the calls to kafka to publish properly captured ?

[VVK (VVetsa)]

Thank you for quick reply.

EDOT Java agent was on my radar. I shall expedite my switch to EDOT java agent.

As of now, these three operations are getting executed (currently with Elastic APM Java agent ) in a sequential order.

I was hoping at least Mongo communication would be captured with Elastic APM Java agent. But that is okay.

Thank you for quick reply @Sylvain_Juge

[VVK (VVetsa)]

@Sylvain_Juge

well... one observation, this can be brought to your dev team.

Why?
My kafka consumer is of type batch=false; meaning, I can not process the messages that are coming to consumer in batch mode (effectively).

Yes, one can say batch=true and process the messages one by one. And, I tried this and I am able to see auto-instrumentation kicking in .

But, behaviour remained the same, it did capture only grpc call. And no dependency capturing of mongo or kafka.

We will definitely go with EDOT, but the path to that approach is long one as all services need to move to it.