We are currently using elastic agent to collect kafka consumer group metric. After a consumer group was configured to use the new consumer rebalance protocol available in Kafka 4.0 , the agent was not collecting the metric for this kafka consumer group.
Current agent version being used is 9.2.4.
I checked the documentation if the agent is able to collect metric for consumer groups using the new rebalance protocol, I could not find any reference in the docuementation.
I also checked that sarama started supporting this feature only from 1.61.0 (link, link), check for kip-848). And it seems the sarama included with elastic agent is older one (sarama v1.46.3, as I can infer from GitHub - elastic/sarama: Sarama is a Go library for Apache Kafka 0.8, and up. · GitHub ).
Even the otel collector version is older one, only supporting kafka 2.x and kafka 3.x.
I tried to do investigation but I might be completely wrong.
Question: Is the consumer group using the kip-848 rebalance protocol supported by elastic agent? If not, when is it planned to be supported?