Filebeat not pushing events to multiple Kafka broker in a single Kafka cluster

[Hari_Prasad]

The Filebeat in my setup pushes the events to a Kafka cluster with 2 brokers. I have added only one node in the host list but both the brokers in the cluster were discovered. I understood this from the Filebeat logs.

But, even though both the brokers were discovered the events are published to only broker.

Below is teh filebeat config for output

 output.kafka:
  hosts: ["host1:port"]
  topic: '%{[fields.document_type]}'
  worker: 4
  partition.round_robin:
    reachable_only: true
  required_acks: 1
  compression: gzip
  compression_level: 3

Even in the logs it lists that it is connected to only one of the broker as registered one.

2019-01-07T06:12:38.789-0800    INFO    kafka/log.go:53 client/metadata fetching metadata for all topics from broker host1:port

2019-01-07T06:12:38.799-0800    INFO    kafka/log.go:53 Connected to broker at host1:port (unregistered)

2019-01-07T06:12:38.806-0800    INFO    kafka/log.go:53 client/brokers registered new broker #1 at host1:port
2019-01-07T06:12:38.806-0800    INFO    kafka/log.go:53 client/brokers registered new broker #0 at host2:port
2019-01-07T06:12:38.806-0800    INFO    kafka/log.go:53 kafka message: Successfully initialized new client
2019-01-07T06:12:38.806-0800    INFO    pipeline/output.go:105  Connection to kafka(host1:port,host2:port) established
2019-01-07T06:12:38.807-0800    INFO    kafka/log.go:53 kafka message: Producer.MaxMessageBytes must be smaller than MaxRequestSize; it will be ignored.
2019-01-07T06:12:38.808-0800    INFO    kafka/log.go:53 producer/broker/0 starting up

2019-01-07T06:12:38.814-0800    INFO    kafka/log.go:53 Connected to broker at host2:port (registered as #0)

The zookeeper console list both the broker in cluster so the Kafka cluster are fine too.

I am not able to figure the mistake that is causing the Filebeat to push to only one Kafka broker.

Thank you

[Hari_Prasad]

I think I found the reason for this. My Kafka cluster was on default setting which was set to one partition. Also I started Kafka cluster with one node when the topic was created. Hence all the topic's partition was allocated to one node. When the 2nd node was added Kafka do not automatically rebalance the partitions across the nodes causing all partition in one place .Hence Filebeat was sending data to only one node.