Kafka establishes tls1.2 with openssl, but not filebeat

Elastic Stack Beats
1

Filebeat is configured to talk to kafka over tls, but it fails with "INFO kafka/log.go:36 Failed to connect to broker : tls: first record does not look like a TLS handshake"

openssl s_client, using the same key, ca and cert, is able to connect.

Wireshark shows "Ignored Unknown Record" where the openssl connection has "Server Hello".

Relevant version info:

  • go1.10 linux/amd64
  • filebeat 7.0.0-alpha1 (amd64), libbeat 7.0.0-alpha1 [unknown built unknown]
  • openssl 1.0.2g
  • kafka 2.12-1.0.0
2

Kafka requires some connection bootstrapping protocol. The bootstrapping first gets the cluster metadata from one of the brokers first. The meta-data is used to connect to the brokers. Check the kafka brokers advertised listeners actually configuring TLS.

Just using openssl s_client is only testing the initial connection during the bootstrap. The final step of actually connecting to the clusters is not simulated by openssl s_client.

3

Wireshark suggests the initial connection is failing.

Does a TLS connection happen before broker.GetMetadata (github.com/elastic/beats/vendor/github.com/Shopify/sarama/client.go:633) ? Further up the callstack is client.Connect() in github.com/elastic/beats/libbeat/outputs/kafka/client.go:72

4

Looks related to this https://issues.apache.org/jira/browse/KAFKA-3647?focusedCommentId=15270520&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-15270520

5

What exactly is wireshark suggesting? Can you share the pcap of filebeat trying to connect to kafka?

Can you share filebeat/kafka configs and logs?

How did you create your certificates?

6

Wireshark is suggesting kafka doesn't respond to filebeat's Client Hello with a Server Hello.

My issue is resolved now, but for future reference the problem was that kafka didn't use the same ciphers as filebeat.

"When creating the keystore on the server, you need to pass the -keyalg RSA argument or else the Kafka broker will operate using an extremely limited set of ciphers, none of which are supported by Golang. See #643 for more details."
(https://github.com/Shopify/sarama/wiki/Frequently-Asked-Questions#why-cant-sarama-connect-to-my-kafka-cluster-using-ssl)

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.