Filebeat 7.11 is not publishing application log into AWS MSK-2.2.1, getting error kafka: client has run out of available brokers to talk

Hi All,
I am unable to post data from filebeat agent to AWS MSK 2.2.1 . Initially I was used filebeat version 7.10.0.As per the elastic team advice I have upgraded into 7.11.0.

I used TLS configuration and please find the configuration below.

I was used AWS CLI to create a topic first and able to produce and consume data successfully.

I used the same TLS configuration in filebeat yml file and used same topic name and broker details.After starting the agent, Its showing error like “ client has run out of available brokers to talk to”.

To confirm either all the brokers are available or not. I used telnet command and able to connect from client machine.

I am having doubt on JKS file, either do we use it or we have to provide pem file for TLS encryption? please suggest.

Topic :-
./kafka-console-producer.sh --broker-list b-1.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094,b-2.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094,b-3.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094 --producer.config client.properties --topic sktopic

Producer:-
[root@ip-172-31-4-58 bin]# ./kafka-console-producer.sh --broker-list b-1.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094,b-2.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094,b-3.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094 --producer.config client.properties --topic sktopic

hi
hello
how r u ?

Consumer :-

[ec2-user@ip-172-31-4-58 software] cd kafka_2.12-2.2.1/ [ec2-user@ip-172-31-4-58 kafka_2.12-2.2.1] cd bin
[ec2-user@ip-172-31-4-58 bin]$ ./kafka-console-consumer.sh --bootstrap-server b-1.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094,b-2.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094,b-3.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094 --consumer.config client.properties --topic sktopic --from-beginning
hi
hello
how r u ?

filebeat.yml :-
#-------------------------------- Kafka Output --------------------------------
output.kafka:

Boolean flag to enable or disable the output module.

enabled: true

The list of Kafka broker addresses from which to fetch the cluster metadata.

The cluster metadata contain the actual Kafka brokers events are published

to.

hosts: [ "b-1.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094",
"b-2.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094",
"b-3.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094" ]

The Kafka topic used for produced events. The setting can be a format string

using any event field. To set the topic from document type use %{[type]}.

topic: "sktopic"

Use TLS settings for encryption

tls.certificate_authorities: ["/tmp/kafka.client.truststore.jks"]

Use SSL settings for HTTPS.

#ssl.enabled: false

Error :-filebeat -7.11.0

2021-02-15T16:31:50.801Z DEBUG [input] input/input.go:139 Run input
2021-02-15T16:31:50.802Z DEBUG [input] log/input.go:205 Start next scan
2021-02-15T16:31:50.802Z DEBUG [input] log/input.go:439 Check file for harvesting: /var/tmp/dummy_log_4.log
2021-02-15T16:31:50.802Z DEBUG [input] log/input.go:530 Update existing file for harvesting: /var/tmp/dummy_log_4.log, offset: 3490
2021-02-15T16:31:50.802Z DEBUG [input] log/input.go:583 Harvester for file is still running: /var/tmp/dummy_log_4.log
2021-02-15T16:31:50.802Z DEBUG [input] log/input.go:226 input states cleaned up. Before: 1, After: 1, Pending: 0
2021-02-15T16:31:55.807Z DEBUG [harvester] log/log.go:107 End of file reached: /var/tmp/dummy_log_4.log; Backoff now.
2021-02-15T16:31:56.873Z ERROR [kafka] kafka/client.go:317 Kafka (topic=sktopic): kafka: client has run out of available brokers to talk to (Is your cluster reachable?)
^C
[root@ip-172-31-4-58 filebeat]# telnet b-1.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com 9094
Trying 172.31.17.54...
Connected to b-1.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com.
Escape character is '^]'.
^CConnection closed by foreign host.
[root@ip-172-31-4-58 filebeat]# telnet b-2.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com 9094
Trying 172.31.44.120...
Connected to b-2.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com.
Escape character is '^]'.
^CConnection closed by foreign host.
[root@ip-172-31-4-58 filebeat]# telnet b-3.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com 9094
Trying 172.31.5.246...
Connected to b-3.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com.
Escape character is '^]'.