Metricbeat kafka module kerberos authentication

[drewkrrb (Kyle)]

Hi
Am trying to enable metricbeat kafka module in my kafka cluster. My kafka cluster uses kerberos auth to access all the topics. Here is my current config:

# Module: kafka
# Docs: https://www.elastic.co/guide/en/beats/metricbeat/7.10/metricbeat-module-kafka.html

# Kafka metrics collected using the Kafka protocol
- module: kafka
  #metricsets:
  #  - partition
  #  - consumergroup
  period: 10s
  hosts: ["localhost:9092"]

  client_id: metricbeat
  #retries: 3
  #backoff: 250ms

  # List of Topics to query metadata for. If empty, all topics will be queried.
  #topics: []

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

  # SASL authentication
  #username: ""
  #password: ""
  kerberos.auth_type: keytab
  kerberos.config_path: /etc/krb5.conf
  kerberos.keytab: /etc/metricbeat/certs/ex_elk.keytab
  kerberos.service_name: kafka
  kerberos.realm: NONPROD.REALM
# Metrics collected from a Kafka broker using Jolokia
- module: kafka
  metricsets:
    - broker
    - producer
    - consumer
  period: 10s
  hosts: ["localhost:8778"]

Am wondering if this is supported using kerberos authentication for kafka module since I don't see it in the document guides.

[Kaiyan_Sheng]

Hello! I think this is added by https://github.com/elastic/beats/pull/16871. Is this what you are looking for? Thanks!

[drewkrrb (Kyle)]

Hi thank you for replying

Yes I've done that if you can see my config above.
But am getting this error:

2021-02-24T14:56:59.342+0800    INFO    module/wrapper.go:259   Error fetching data for metricset kafka.consumergroup: error in connect: getting cluster client for advertised broker with address 10.244.17.22:9092: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)
2021-02-24T14:57:09.414+0800    INFO    module/wrapper.go:259   Error fetching data for metricset kafka.consumergroup: error in connect: getting cluster client for advertised broker with address 10.244.17.22:9092: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

I've test the keytab file and am able to get all the topics.

[Kaiyan_Sheng]

@kvch Do you know how to fix this issue? TIA!!

[kvch (Noémi Ványi)]

Unfortunately, Kerberos authentication is not yet added to kafka module in Metricbeat.