Filebeat cannot connect Kafka TLS

Elastic Stack Beats
1

Dear Elastic team,

Test Environment: Local zookeeper, Kafka and Filebeat in a machine (kafka0).
Kafka version: 2.13-2.4.0
Filebeat version: 7.6

I tried to setup Filebeat to connect Kafka with TLS option.

I tested this command and the result is OK.

openssl s_client -connect localhost:9093 -tls1_2 -CAfile ca-chain.pem -cert producer.pem -key producer-key.pem

New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: CFB7D9FD3E646C03FD5AC44F05F3AFD4DA6BD4D73DC9ABDD3E3BE176CD27B1CD
Session-ID-ctx:
Master-Key: C08D857BB60DD1BD6ABBD9058D62005C5C06DF6964F36AE7CFC4998A91A097997A71638C99C6CDD38A3E5719EF9EEC4C
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
Start Time: 1583859439
Timeout : 7200 (sec)
Verify return code: 0 (ok)

However, it doesn't work when I configured in Filebeat.

Mar 10 17:01:39 kafka0 filebeat[8653]: 2020-03-10T17:01:39.636Z INFO kafka/log.go:53 kafka message: client/metadata no available broker to send metadata request to
Mar 10 17:01:39 kafka0 filebeat[8653]: 2020-03-10T17:01:39.636Z INFO kafka/log.go:53 client/brokers resurrecting 1 dead seed brokers
Mar 10 17:01:39 kafka0 filebeat[8653]: 2020-03-10T17:01:39.636Z INFO kafka/log.go:53 client/metadata retrying after 250ms... (3 attempts remaining)
Mar 10 17:01:39 kafka0 filebeat[8653]: 2020-03-10T17:01:39.902Z INFO kafka/log.go:53 client/metadata fetching metadata for [test] from broker kafka0:9093
Mar 10 17:01:39 kafka0 filebeat[8653]: 2020-03-10T17:01:39.909Z INFO kafka/log.go:53 Failed to connect to broker kafka0:9093: x509: certificate is not valid for any names, but wanted to match kafka0
Mar 10 17:01:39 kafka0 filebeat[8653]: 2020-03-10T17:01:39.909Z INFO kafka/log.go:53 client/metadata got error from broker -1 while fetching metadata: x509: certificate is not valid for any names, but wanted to match kafka0

My Filebeat configuration.

output.kafka:
  # initial brokers for reading cluster metadata
  hosts: ["kafka0:9093"]

  # message topic selection + partitioning
  topic: test
  partition.round_robin:
reachable_only: false

  required_acks: 1
  compression: gzip
  max_message_bytes: 1000000
  ssl:
certificate: '/etc/filebeat/producer.pem'
key: '/etc/filebeat/producer-key.pem'
certificate_authorities:
  - '/etc/filebeat/ca-chain.pem'

The ca-chain.pem contains intermediate-ca and ca certificate.

I follow the instruction in this thread.

Thanks

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.