Filebeat + Kafka error

[kamandohl (Carl)]

I have Googled enough, but unable to find the error I am currently seeing.

Filebeat - 5.6.4 and 6.0.0
Kafka - 10.1

2017/11/27 13:10:03.602226 log.go:16: WARN kafka message: Initializing new client
2017/11/27 13:10:03.602342 log.go:12: WARN client/metadata fetching metadata for all topics from broker localhost
2017/11/27 13:10:03.603931 log.go:12: WARN Connected to broker at localhost (unregistered)
2017/11/27 13:10:03.605022 log.go:16: WARN kafka message: client/metadata got error from broker while fetching metadata:%!(EXTRA *errors.errorString=unexpected EOF)
2017/11/27 13:10:03.605074 log.go:12: WARN Closed connection to broker localhost
2017/11/27 13:10:03.611668 log.go:12: WARN client/brokers resurrecting 3 dead seed brokers
2017/11/27 13:10:03.611682 log.go:12: WARN client/metadata retrying after 250ms... (3 attempts remaining)

filebeat.yml

filebeat.prospectors:

type: log
enabled: true
- /tmp/test.log

output.kafka:
enabled: true
hosts: ["localhost1:9093","localhost2","localhost3:9093"]
topic: '%{[type]}'
ssl:
ssl.certificate_authorities: "ca.pem"
ssl.certificate: : "cert.pem"
ssl.key: "key.pem"
key_passphrase: *****

[steffens (Steffen Siering)]

Please format logs and configs using the </>-button to make your post more readable (configs are sensitive to indentation, without formatting it's hard to check configs being correct).

1. The broker is closing the connection. Have you check the kafka logs?

2. The SSL/TLS settings must be configured via ssl namespace, not tls. Looks like you the beat is trying plain text connection, while kafka is expecting a SSL/TLS handshake.

3. remove duplicates from hosts setting. As you only have one broker, remove the others. The hosts setting is required for cluster connection bootstrapping only (get metadata to find all nodes in a cluster).

4. From you configs it looks like you want to have client authentication. But I'm missing the CA configuration for accepting the servers certificates.

Can you try without TLS first? Then add TLS server authentication and finally enable client authentication? In general properly setting up (or worse debugging) TLS can be somewhat difficult at times, better do it step by step.