Metricbeat Kafka module won’t work with Kafka 4.0.0

Elastic Stack Beats
1

Hello, everyone.

I see this logs when trying to connect to Kafka 4.0.0 with Metricbeat 9.0.0 kafka module.

  • Metricbeat log
{"log.level":"warn","@timestamp":"2025-05-22T16:25:44.754+0900","log.logger":"kafka.partition","log.origin":{"function":"github.com/elastic/beats/v7/metricbeat/module/kafka/partition.(*MetricSet).Fetch","file.name":"partition/partition.go","file.line":129},"message":"failed to query kafka partition (__consumer_offsets:38) offsets: failed to get oldest offset: get available offsets failed: write tcp [::1]:51096->[::1]:9092: write: broken pipe","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2025-05-22T16:25:44.754+0900","log.logger":"kafka.partition","log.origin":{"function":"github.com/elastic/beats/v7/metricbeat/module/kafka/partition.(*MetricSet).Fetch","file.name":"partition/partition.go","file.line":129},"message":"failed to query kafka partition (eai_topic:2) offsets: failed to get oldest offset: get available offsets failed: write tcp [::1]:51096->[::1]:9092: write: broken pipe","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2025-05-22T16:25:44.754+0900","log.logger":"kafka.partition","log.origin":{"function":"github.com/elastic/beats/v7/metricbeat/module/kafka/partition.(*MetricSet).Fetch","file.name":"partition/partition.go","file.line":129},"message":"failed to query kafka partition (eai_topic:0) offsets: failed to get oldest offset: get available offsets failed: write tcp [::1]:51096->[::1]:9092: write: broken pipe","service.name":"metricbeat","ecs.version":"1.6.0"}
  • Kafka log
[2025-05-21 19:50:48,503] ERROR Closing socket for 0:0:0:0:0:0:0:1:9092-0:0:0:0:0:0:0:1:64441-0-0 because of error (kafka.network.Processor)
org.apache.kafka.common.errors.UnsupportedVersionException: Received request for api with key 2 (ListOffsets) and unsupported version 0
[2025-05-21 19:50:48,508] ERROR Exception while processing request from 0:0:0:0:0:0:0:1:9092-0:0:0:0:0:0:0:1:64441-0-0 (kafka.network.Processor)
org.apache.kafka.common.errors.UnsupportedVersionException: Received request for api with key 2 (ListOffsets) and unsupported version 0
[2025-05-21 19:51:11,896] ERROR Closing socket for 0:0:0:0:0:0:0:1:9092-0:0:0:0:0:0:0:1:64468-1-2 because of error (kafka.network.Processor)
org.apache.kafka.common.errors.UnsupportedVersionException: Received request for api with key 2 (ListOffsets) and unsupported version 0

I think the cause is kafka module is using removed api version that Kafka 4.0.0 doesn't support anymore.
Here's a similar issue I found.

Sorry for creating an issueon github before creating topic on discuss.
This is the issue that I created.