FileBeats -> MSK : SSL handshake failed when TLS is enabled

Hi All..
I have created a MSK cluster with 3 brokers and enabled encryption at rest and in transit but
disabled TLS authentication.
I'm trying to send data from Filebeat (7.10.0) --> AWS MSK over TLS endpoint (9094).
Filebeat in installed in one of the AWS EC2 instance within same VPC of AWS MSK cluster.

Also included the below value in filebeat.yml o/p section:
tls.certificate_authorities: ["/tmp/kafka.client.truststore.jks"]

We have generated the kafka.client.truststore.jks by coping from Java . Refer: [+] Step 6: Produce and Consume Data - Step 6: Produce and Consume Data - Amazon Managed Streaming for Apache Kafka

From Filebeat Log I'm getting the below msg:
2021-02-12T14:22:33.888Z DEBUG kafka/client.go:291 Kafka publish failed with: circuit breaker is open

I also enabled cloud watch logs for the broker and on the broker side I see the following error message "SocketServer brokerId=1 Failed authentication with /INTERNAL_IP (SSL handshake failed) ("

Could you please help out here, many thanks?


Duplicate FileBeats -> MSK : SSL handshake failed when TLS is enabled

Hi All,

We have fixed this issue :slight_smile: - adding here for the benefits of others (if).

We resolved the SSL handshake issue in MSK end by adding the following entries in filebeat config file.
ssl.enabled: true
tls: certificate_authorities - "/home/ec2-user/kafka.client.truststore.jks"

Alternatively, we can convert the .jks into .pem and provide the below params in filebeat config file:
ssl.certificate_authorities: ["/home/ec2-user/certificatefile.pem"]

By doing anyone of the above we are able to successfully write and read TLS encrypted data from AWS MSK.


This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.