Hi All,
I am unable to post data from filebeat agent to AWS MSK 2.2.1 . Initially I was used filebeat version 7.10.0.As per the elastic team advice I have upgraded into 7.11.0.
I used TLS configuration and please find the configuration below.
I was used AWS CLI to create a topic first and able to produce and consume data successfully.
I used the same TLS configuration in filebeat yml file and used same topic name and broker details.After starting the agent, Its showing error like “ client has run out of available brokers to talk to”.
To confirm either all the brokers are available or not. I used telnet command and able to connect from client machine.
I am having doubt on JKS file, either do we use it or we have to provide pem file for TLS encryption? please suggest.
Topic :-
./kafka-console-producer.sh --broker-list b-1.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094,b-2.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094,b-3.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094 --producer.config client.properties --topic sktopic
Producer:-
[root@ip-172-31-4-58 bin]# ./kafka-console-producer.sh --broker-list b-1.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094,b-2.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094,b-3.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094 --producer.config client.properties --topic sktopic
hi
hello
how r u ?
Consumer :-
[ec2-user@ip-172-31-4-58 software]$ cd kafka_2.12-2.2.1/
[ec2-user@ip-172-31-4-58 kafka_2.12-2.2.1]$ cd bin
[ec2-user@ip-172-31-4-58 bin]$ ./kafka-console-consumer.sh --bootstrap-server b-1.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094,b-2.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094,b-3.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094 --consumer.config client.properties --topic sktopic --from-beginning
hi
hello
how r u ?
filebeat.yml :-
#-------------------------------- Kafka Output --------------------------------
output.kafka:
Boolean flag to enable or disable the output module.
enabled: true
The list of Kafka broker addresses from which to fetch the cluster metadata.
The cluster metadata contain the actual Kafka brokers events are published
to.
hosts: [ "b-1.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094",
"b-2.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094",
"b-3.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com:9094" ]
The Kafka topic used for produced events. The setting can be a format string
using any event field. To set the topic from document type use %{[type]}.
topic: "sktopic"
Use TLS settings for encryption
tls.certificate_authorities: ["/tmp/kafka.client.truststore.jks"]
Use SSL settings for HTTPS.
#ssl.enabled: false
Error :-filebeat -7.11.0
2021-02-15T16:31:50.801Z DEBUG [input] input/input.go:139 Run input
2021-02-15T16:31:50.802Z DEBUG [input] log/input.go:205 Start next scan
2021-02-15T16:31:50.802Z DEBUG [input] log/input.go:439 Check file for harvesting: /var/tmp/dummy_log_4.log
2021-02-15T16:31:50.802Z DEBUG [input] log/input.go:530 Update existing file for harvesting: /var/tmp/dummy_log_4.log, offset: 3490
2021-02-15T16:31:50.802Z DEBUG [input] log/input.go:583 Harvester for file is still running: /var/tmp/dummy_log_4.log
2021-02-15T16:31:50.802Z DEBUG [input] log/input.go:226 input states cleaned up. Before: 1, After: 1, Pending: 0
2021-02-15T16:31:55.807Z DEBUG [harvester] log/log.go:107 End of file reached: /var/tmp/dummy_log_4.log; Backoff now.
2021-02-15T16:31:56.873Z ERROR [kafka] kafka/client.go:317 Kafka (topic=sktopic): kafka: client has run out of available brokers to talk to (Is your cluster reachable?)
^C
[root@ip-172-31-4-58 filebeat]# telnet b-1.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com 9094
Trying 172.31.17.54...
Connected to b-1.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com.
Escape character is '^]'.
^CConnection closed by foreign host.
[root@ip-172-31-4-58 filebeat]# telnet b-2.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com 9094
Trying 172.31.44.120...
Connected to b-2.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com.
Escape character is '^]'.
^CConnection closed by foreign host.
[root@ip-172-31-4-58 filebeat]# telnet b-3.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com 9094
Trying 172.31.5.246...
Connected to b-3.sk-kafka-dev.vp11l7.c5.kafka.eu-west-1.amazonaws.com.
Escape character is '^]'.