Metricbeat to collect metrics from Kerberised Kafka + SSLCa configuration

[lskaa (Derek)]

Hi All,

May I know if anyone has experience to configure Metricbeat to collect metrics from Kerberised Kafka?

Here is my kafka.yml. I also add Read and Describe operations to my Kafka instances for a user stats.

• module: kafka
  metricsets:
  • partition
  • consumergroup
    period: 10s
    hosts:
  • broker1:9092
  • broker2:9092
  • broker3:9092
    client_id: stats

According to the log, it connected to the brokers seemingly.

2020-01-10T09:28:57.157+1300 INFO kafka/log.go:53 Connected to broker at broker1:9092 (unregistered)
2020-01-10T09:28:57.390+1300 INFO kafka/log.go:53 Closed connection to broker broker1:9092

However, it failed to fetch data for the partition and the consumergroup.

2020-01-10T09:28:57.672+1300 INFO module/wrapper.go:252 Error fetching data for metricset kafka.partition: error in connect: failed to query metadata: EOF
2020-01-10T09:28:57.672+1300 INFO module/wrapper.go:252 Error fetching data for metricset kafka.consumergroup: error in connect: failed to query metadata: EOF

Thank you

[olorasde (oscar)]

Hi! I'm trying to configure metricbeat with kafka and i have exactly the same error than you.

I'm using kafka 2.1.1 and metricbeat 7.4.2. I suspect that the problem maybe is related to version og metricbeat.

Now im trying to deploy 7.5.0 version and check.

I'll write the results here.

[olorasde (oscar)]

seem that metricbeat 7.5.1 has a bug trying to activate kafka module.

Its possible?

[lskaa (Derek)]

It seems to me not the issue of the module. It looks to me that the issue of Metricbeat collecting metrics from Kerberised Kafka as I've found from other post.

Metricbeat not able to collect metrics from Kerberised Kafka

Seemingly, there is not much update.

[olorasde (oscar)]

If i undertand correcty, this seems that kafka securized cant allow be monitorizet by metricbeat? its a kafka issue?

[lskaa (Derek)]

I can query the metadata via the scripts from Kafka using the same client id as in the kafka.yml file of metricbeat. However, metricbeat complained query metadata issue. So it seems to me it's more on metricbeat issue rather than the Kerberised Kafka.