Metricbeats kafka module is not using SSL

Elastic Observability Metrics
1

Hi,

I'm trying to set up the elk monitoring for my kafka cluster using metricbeat kafka module.

My kafka cluster is using SASL_SSL security protocol. I have configured my kafka.yml in metricbeat like below

  • module: kafka
    metricsets: ["partition", "consumergroup", "broker"]
    period: 10s
    hosts: ["host:port"]
    security.protocol: "SASL_SSL"
    sasl.mechanism: "PLAIN"
    ssl.verification_mode: none
    ssl.enabled: true
    ssl.certificate_authorities: ["ca.pem"]
    ssl.certificate: "mycert.pem"
    ssl.key: "mykey.key"

I have verified the connectivity using openssl with above key and cert and SSL level connectivity is okay through openssl. However, when my metricbeat is connecting to Kafka, kafka broker is complaining as below

(errorMessage=SSL handshake failed caused by Unrecognized SSL message, plaintext connection?) (org.apache.kafka.common.network.Selector)
(errorMessage=Unexpected Kafka request of type METADATA during SASL handshake.) (org.apache.kafka.common.network.Selector)

This looks like metricbeat client is trying to connect without using SSL.

Appreciate any help to resolve this.

Thanks in advance!

I'm using ELK and metricbeat version 7.17

2

Hello, I believe you are missing the SASL part of the configuration

# SASL authentication
  #username: ""
  #password: ""

  # SASL authentication mechanism used. Can be one of PLAIN, SCRAM-SHA-256 or SCRAM-SHA-512.
  # Defaults to PLAIN when `username` and `password` are configured.
  #sasl.mechanism: ''

There should be a user for it.

Please check Here

Regards.

3

Thanks for the reply @coezdemir

I have added username and password fields, still getting below error in kafka broker side

(errorMessage=SSL handshake failed caused by Unrecognized SSL message, plaintext connection?) (org.apache.kafka.common.network.Selector)

And below error in metricbeat side

Error fetching data for metricset kafka.broker: error making http request: Post "http://<host>:<port>/jolokia/%3FignoreErrors=true&canonicalNaming=false": net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x15\x03\x03\x00\x02\x02P"

Is it expected that metricbeat is using http in this case?

Updated metricbeat kafka yml file is as below

- module: kafka
  metricsets: ["partition", "consumergroup", "broker"]
  period: 10s
  hosts: ["<host>:<port>"] 
  sasl.mechanism: "PLAIN"
  username: "<user>"
  password: "<pwd>"
  ssl.verification_mode: none
  ssl.enabled: true
  sasl.enabled: true
  ssl.certificate_authorities: ["ca.pem"]
  ssl.certificate: "mycert.pem"
  ssl.key: "mykey.key"
  bulk_max_size: 50
4

Hello,

Could you please share how you are starting the kafka, and which ports you are giving to your metricbeats kafka modules.

As far as I can see, the modules "broker" scrape uses Jolokai agent. So your port has to hit Jolokai port, not the kafka external listener port.

# Kafka metrics collected using the Kafka protocol
- module: kafka
  #metricsets:
  #  - partition
  #  - consumergroup
  period: 10s
  hosts: ["localhost:9092"]

From documentation example config, it can be seen that only "partition" and "consumergroup" metricset is available to be collected from kafkas external port.

# Metrics collected from a Kafka broker using Jolokia
#- module: kafka
#  metricsets:
#    - broker
#  period: 10s
#  hosts: ["localhost:8779"]

The "broker" metric set needs a Jolokai agent port as shown above.

Regards.