FileBeats -> MSK : SSL handshake failed when TLS is enabled

[Ravi342883 (RAVINDRAN SHANMUGAM)]

Hi All..
I have created a MSK cluster with 3 brokers and enabled encryption at rest and in transit but
disabled TLS authentication.
I'm trying to send data from Filebeat (7.10.0) --> AWS MSK over TLS endpoint (9094).
Filebeat in installed in one of the AWS EC2 instance within same VPC of AWS MSK cluster.

Also included the below value in filebeat.yml o/p section:
tls.certificate_authorities: ["/tmp/kafka.client.truststore.jks"]

We have generated the kafka.client.truststore.jks by coping from Java . Refer: [+] Step 6: Produce and Consume Data - Step 6: Produce and Consume Data - Amazon Managed Streaming for Apache Kafka

From Filebeat Log I'm getting the below msg:
2021-02-12T14:22:33.888Z DEBUG https://forums.aws.amazon.com/ kafka/client.go:291 Kafka publish failed with: circuit breaker is open

I also enabled cloud watch logs for the broker and on the broker side I see the following error message "SocketServer brokerId=1 Failed authentication with /INTERNAL_IP (SSL handshake failed) (org.apache.kafka.common.network.Selector)"

Could you please help out here, many thanks?

Thanks,
Ravi
Ravindharan@gmail.com

[Mario_Castro]

Duplicate FileBeats -> MSK : SSL handshake failed when TLS is enabled

[Ravi342883 (RAVINDRAN SHANMUGAM)]

Hi All,

We have fixed this issue - adding here for the benefits of others (if).

We resolved the SSL handshake issue in MSK end by adding the following entries in filebeat config file.
ssl.enabled: true
tls: certificate_authorities - "/home/ec2-user/kafka.client.truststore.jks"

Alternatively, we can convert the .jks into .pem and provide the below params in filebeat config file:
ssl.certificate_authorities: ["/home/ec2-user/certificatefile.pem"]

By doing anyone of the above we are able to successfully write and read TLS encrypted data from AWS MSK.

Thanks,
Ravi
Ravindharan@gmail.com