# Filebeat not pushing events to multiple Kafka broker in a single Kafka cluster

**URL:** <https://discuss.elastic.co/t/filebeat-not-pushing-events-to-multiple-kafka-broker-in-a-single-kafka-cluster/163250>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [January 7, 2019, 5:16pm UTC](https://discuss.elastic.co/t/filebeat-not-pushing-events-to-multiple-kafka-broker-in-a-single-kafka-cluster/163250 "2019-01-07T17:16:01Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Hari\_Prasad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hari_prasad/32/56784_2.png) [@Hari\_Prasad](https://discuss.elastic.co/u/Hari_Prasad)\
**Post date:** [January 7, 2019, 5:16pm UTC](https://discuss.elastic.co/t/filebeat-not-pushing-events-to-multiple-kafka-broker-in-a-single-kafka-cluster/163250/1 "2019-01-07T17:16:02Z")

</div>

The Filebeat in my setup pushes the events to a Kafka cluster with 2 brokers. I have added only one node in the host list but both the brokers in the cluster were discovered. I understood this from the Filebeat logs.

But, even though both the brokers were discovered the events are published to only broker.

Below is teh filebeat config for output

```
 output.kafka:
  hosts: ["host1:port"]
  topic: '%{[fields.document_type]}'
  worker: 4
  partition.round_robin:
    reachable_only: true
  required_acks: 1
  compression: gzip
  compression_level: 3

```

Even in the logs it lists that it is connected to only one of the broker as registered one.

```
2019-01-07T06:12:38.789-0800 INFO kafka/log.go:53 client/metadata fetching metadata for all topics from broker host1:port

2019-01-07T06:12:38.799-0800 INFO kafka/log.go:53 Connected to broker at host1:port (unregistered)

2019-01-07T06:12:38.806-0800 INFO kafka/log.go:53 client/brokers registered new broker #1 at host1:port
2019-01-07T06:12:38.806-0800 INFO kafka/log.go:53 client/brokers registered new broker #0 at host2:port
2019-01-07T06:12:38.806-0800 INFO kafka/log.go:53 kafka message: Successfully initialized new client
2019-01-07T06:12:38.806-0800 INFO pipeline/output.go:105 Connection to kafka(host1:port,host2:port) established
2019-01-07T06:12:38.807-0800 INFO kafka/log.go:53 kafka message: Producer.MaxMessageBytes must be smaller than MaxRequestSize; it will be ignored.
2019-01-07T06:12:38.808-0800 INFO kafka/log.go:53 producer/broker/0 starting up

2019-01-07T06:12:38.814-0800 INFO kafka/log.go:53 Connected to broker at host2:port (registered as #0)

```

The zookeeper console list both the broker in cluster so the Kafka cluster are fine too.

I am not able to figure the mistake that is causing the Filebeat to push to only one Kafka broker.

Thank you

---

<div class="post-metadata">

**Author:** ![Hari\_Prasad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hari_prasad/32/56784_2.png) [@Hari\_Prasad](https://discuss.elastic.co/u/Hari_Prasad)\
**Post date:** [January 14, 2019, 3:33am UTC](https://discuss.elastic.co/t/filebeat-not-pushing-events-to-multiple-kafka-broker-in-a-single-kafka-cluster/163250/2 "2019-01-14T03:33:20Z")

</div>

I think I found the reason for this. My Kafka cluster was on default setting which was set to one partition. Also I started Kafka cluster with one node when the topic was created. Hence all the topic's partition was allocated to one node. When the 2nd node was added Kafka do not automatically rebalance the partitions across the nodes causing all partition in one place .Hence Filebeat was sending data to only one node.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 11, 2019, 3:33am UTC](https://discuss.elastic.co/t/filebeat-not-pushing-events-to-multiple-kafka-broker-in-a-single-kafka-cluster/163250/3 "2019-02-11T03:33:30Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.