# Filebeat 7.11 is not publishing application log into AWS MSK-2.2.1, getting error kafka: client has run out of available brokers to talk **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [February 15, 2021, 5:27pm UTC](https://discuss.elastic.co/t/filebeat-7-11-is-not-publishing-application-log-into-aws-msk-2-2-1-getting-error-kafka-client-has-run-out-of-available-brokers-to-talk/264339 "2021-02-15T17:27:55Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![saroja](https://avatars.discourse-cdn.com/v4/letter/s/eada6e/32.png) [@saroja](https://discuss.elastic.co/u/saroja)\ **Post date:** [February 15, 2021, 5:27pm UTC](https://discuss.elastic.co/t/filebeat-7-11-is-not-publishing-application-log-into-aws-msk-2-2-1-getting-error-kafka-client-has-run-out-of-available-brokers-to-talk/264339/1 "2021-02-15T17:27:55Z")
Hi All, I am unable to post data from filebeat agent to AWS MSK 2.2.1 . Initially I was used filebeat version 7.10.0.As per the elastic team advice I have upgraded into 7.11.0. I used TLS configuration and please find the configuration below. I was used AWS CLI to create a topic first and able to produce and consume data successfully. I used the same TLS configuration in filebeat yml file and used same topic name and broker details.After starting the agent, Its showing error like “ client has run out of available brokers to talk to”. To confirm either all the brokers are available or not. I used telnet command and able to connect from client machine. I am having doubt on JKS file, either do we use it or we have to provide pem file for TLS encryption? please suggest. Topic :- ./kafka-console-producer.sh --broker-list [b-1.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094](http://b-1.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094),[b-2.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094](http://b-2.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094),[b-3.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094](http://b-3.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094) --producer.config client.properties --topic sktopic Producer:- [root@ip-172-31-4-58 bin]# ./kafka-console-producer.sh --broker-list [b-1.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094](http://b-1.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094),[b-2.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094](http://b-2.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094),[b-3.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094](http://b-3.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094) --producer.config client.properties --topic sktopic > hi > hello > how r u ? Consumer :- [ec2-user@ip-172-31-4-58 software]$ cd kafka\_2.12-2.2.1/ [ec2-user@ip-172-31-4-58 kafka\_2.12-2.2.1]$ cd bin [ec2-user@ip-172-31-4-58 bin]$ ./kafka-console-consumer.sh --bootstrap-server [b-1.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094](http://b-1.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094),[b-2.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094](http://b-2.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094),[b-3.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094](http://b-3.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094) --consumer.config client.properties --topic sktopic --from-beginning hi hello how r u ? filebeat.yml :- #-------------------------------- Kafka Output -------------------------------- output.kafka: # Boolean flag to enable or disable the output module. enabled: true # The list of Kafka broker addresses from which to fetch the cluster metadata. # The cluster metadata contain the actual Kafka brokers events are published # to. hosts: ["[b-1.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094](http://b-1.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094)", "[b-2.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094](http://b-2.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094)", "[b-3.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094](http://b-3.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094)" ] # The Kafka topic used for produced events. The setting can be a format string # using any event field. To set the topic from document type use `%{[type]}`. topic: "sktopic" # Use TLS settings for encryption tls.certificate\_authorities: ["/tmp/kafka.client.truststore.jks"] # Use SSL settings for HTTPS. #ssl.enabled: false Error :-filebeat -7.11.0 2021-02-15T16:31:50.801Z DEBUG [input] input/input.go:139 Run input 2021-02-15T16:31:50.802Z DEBUG [input] log/input.go:205 Start next scan 2021-02-15T16:31:50.802Z DEBUG [input] log/input.go:439 Check file for harvesting: /var/tmp/dummy\_log\_4.log 2021-02-15T16:31:50.802Z DEBUG [input] log/input.go:530 Update existing file for harvesting: /var/tmp/dummy\_log\_4.log, offset: 3490 2021-02-15T16:31:50.802Z DEBUG [input] log/input.go:583 Harvester for file is still running: /var/tmp/dummy\_log\_4.log 2021-02-15T16:31:50.802Z DEBUG [input] log/input.go:226 input states cleaned up. Before: 1, After: 1, Pending: 0 2021-02-15T16:31:55.807Z DEBUG [harvester] log/log.go:107 End of file reached: /var/tmp/dummy\_log\_4.log; Backoff now. 2021-02-15T16:31:56.873Z ERROR [kafka] kafka/client.go:317 Kafka (topic=sktopic): kafka: client has run out of available brokers to talk to (Is your cluster reachable?) ^C [root@ip-172-31-4-58 filebeat]# telnet [b-1.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com](http://b-1.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com) 9094 Trying 172.31.17.54... Connected to [b-1.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com](http://b-1.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com). Escape character is '^]'. ^CConnection closed by foreign host. [root@ip-172-31-4-58 filebeat]# telnet [b-2.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com](http://b-2.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com) 9094 Trying 172.31.44.120... Connected to [b-2.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com](http://b-2.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com). Escape character is '^]'. ^CConnection closed by foreign host. [root@ip-172-31-4-58 filebeat]# telnet [b-3.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com](http://b-3.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com) 9094 Trying 172.31.5.246... Connected to [b-3.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com](http://b-3.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com). Escape character is '^]'. ---
**Author:** ![Mario\_Castro](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mario_castro/32/35107_2.png) [@Mario\_Castro](https://discuss.elastic.co/u/Mario_Castro)\ **Post date:** [February 16, 2021, 12:00pm UTC](https://discuss.elastic.co/t/filebeat-7-11-is-not-publishing-application-log-into-aws-msk-2-2-1-getting-error-kafka-client-has-run-out-of-available-brokers-to-talk/264339/2 "2021-02-16T12:00:31Z")
---
**Author:** ![Mario\_Castro](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mario_castro/32/35107_2.png) [@Mario\_Castro](https://discuss.elastic.co/u/Mario_Castro)\ **Post date:** [February 16, 2021, 12:00pm UTC](https://discuss.elastic.co/t/filebeat-7-11-is-not-publishing-application-log-into-aws-msk-2-2-1-getting-error-kafka-client-has-run-out-of-available-brokers-to-talk/264339/3 "2021-02-16T12:00:50Z")