TLS Certificates and Kafka TLS Secured

[mschuch (Marcel)]

Hi,
i have three Kafka Brokers configured in Heartime.

    - type: tcp
      id: kafka
      name: Kafka Brokers
      hosts:  ["sl8-0349.xxxx:9092","sl8-0350.xxxxx:9092","sl8-0351.xxxx:9092"]  # default TCP Echo Protocol
      schedule: '@every 5s'
     ssl:
       certificate_authorities: ['/etc/ca.crt']
       supported_protocols: ["TLSv1.0", "TLSv1.1", "TLSv1.2"]

How can i see when TLS Certificates expire and why can Uptime not connect to a SSL port.
The broker says the handshake fails, what do i have to do, to set the SSL settings ? Where do i have to put them in heartbeat?
Metricbeat has specific settings for TLS and works as intended.
Thanks for any help.

[PaulB-Elastic]

Your configuration looks ok. Do your hosts definitely have TLS enabled, on port 9092?
Can you post any errors you are getting as well please.

[mschuch (Marcel)]

This is the error message i get from the Kafka Broker:

[2021-02-16 12:10:14,509] INFO [SocketServer brokerId=1] Failed authentication with /10.10.146.29 (SSL handshake failed) (org.apache.kafka.common.network.Selector)
javax.net.ssl|ERROR|45|data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-2|2021-02-16 12:10:14.509 GMT|TransportContext.java:342|Fatal (BAD_CERTIFICATE): Empty server certificate chain (
"throwable" : {
  javax.net.ssl.SSLHandshakeException: Empty server certificate chain
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:337)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:293)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:284)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:390)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:375)
        at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1074)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1008)
        at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:430)
        at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:514)
        at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:368)
        at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:291)
        at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:173)
        at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:547)
        at org.apache.kafka.common.network.Selector.poll(Selector.java:485)
        at kafka.network.Processor.poll(SocketServer.scala:913)
        at kafka.network.Processor.run(SocketServer.scala:816)
        at java.base/java.lang.Thread.run(Thread.java:834)}

)

[Andrew_Cholakian1 (Andrew Cholakian)]

This seems Kakfka specific, so I don't know how much help we'll be.

I'd like to ask, do you have client cert auth enabled? I'm wondering if Kafka's expecting a client cert and throwing a weird error.

[mschuch (Marcel)]

Yes mutual TLS is activated in Kafka, the broker wants to have a client certificate. In MetricBeat this works pretty well. At the moment I am struggeling to understand what the intention was with the SSL options in Uptime. Does that only work if the server has no mTLS option activated?
And how does heartbeat get the information about the Certificate dates?