Filebeat -> kafka -> logstash multiple brokers

Elastic Stack Logstash
1

Hi,
I have 3 brokers in one kafka cluster with zookeeper server and i create some topics with --replication-factor 3 --partitions 3
I run the zookeeper and the 3 brokers it works fine but when i kill the broker 1 it didn't work it should work with broker 2 or 3 when broker 1 comes back right?
So in this case logstash will wait the broker 1 who works in port 9092 to work there is no solution to work with broker 2 or 3 if broker 1 down ?
filebeat.yml

output.kafka:
  hosts: ["127.0.0.1:9092","127.0.0.1:9093","127.0.0.1:9094"]
  topic: '%{[fields.kafka_topic]}'
  codec.format:
    string: '%{[message]}'
  partition.round_robin:
    reachable_only: false
  required_acks: 1
  compression: gzip
  max_message_bytes: 1000000000
  close_inactive: 50m

logstash.conf

input {
    kafka {
    bootstrap_servers => "127.0.0.1:9092,127.0.0.1:9093,127.0.0.1:9094"
    topics => ["kafka-topic-1","kafka-topic-2","kafka-topic-3","kafka-topic-4"]
    decorate_events => true
  }
}

Is my config correct?

Any help would be sincerely appreciate!

2

And i created the topics with this command .\bin\windows\kafka-topics.bat --create --bootstrap-server localhost:9092 --replication-factor 3 --partitions 3 --topic kafka-topic-1
because this command gives me an error .\bin\windows\kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 3 --partions 3 --topic kafka-topic-1

Exception in thread "main" joptsimple.UnrecognizedOptionException: zookeeper is not a recognized option

Any help would be sincerely appreciate!
Thanks!

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.