# Two logstash Servers with Kafka

**URL:** <https://discuss.elastic.co/t/two-logstash-servers-with-kafka/45238>\
**Category:** Logstash\
**Created:** [March 23, 2016, 2:49pm UTC](https://discuss.elastic.co/t/two-logstash-servers-with-kafka/45238 "2016-03-23T14:49:34Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![kirill\_polishchuk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kirill_polishchuk/32/6489_2.png) [@kirill\_polishchuk](https://discuss.elastic.co/u/kirill_polishchuk)\
**Post date:** [March 23, 2016, 2:49pm UTC](https://discuss.elastic.co/t/two-logstash-servers-with-kafka/45238/1 "2016-03-23T14:49:34Z")

</div>

Hello

I have two logstash servers as front-end, with Kafka Output and two logstash servers as back-end with Kafka Input.  
The front-ends are putting all events into one topic, theoretically the back-ends should read from the topic. In my case only one back-end servers can read from Kafka.  
i was trying to set up Replicated ZooKeeper Cluster and Multi-broker cluster with replicated topic, and still have the same.  
Can you help me, what I am doing wrong ?

Logstash Kafka Output:  
output {  
kafka {  
topic\_id =\> "logstash-logs-replicated"  
broker\_list =\> "[xxx.xxx.xxx.xxx:9092](http://xxx.xxx.xxx.xxx:9092)"  
}  
}

Logstash Kafka Input:  
input {  
kafka {  
topic\_id =\> 'logstash-logs-replicated'  
}  
}

---

<div class="post-metadata">

**Author:** ![Joe\_Lawson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joe_lawson/32/3390_2.png) [@Joe\_Lawson](https://discuss.elastic.co/u/Joe_Lawson)\
**Post date:** [March 24, 2016, 6:18pm UTC](https://discuss.elastic.co/t/two-logstash-servers-with-kafka/45238/2 "2016-03-24T18:18:59Z")

</div>

If you want each backend logstash server to read from the same Kafka topic on its own (ie not in tandem) then you need to set the `group_id` differently on each system. What is happening is the Kafka consumer uses the group\_id to syncronize what is read from an input so if you want two independent readers of a topic, each needs its how group\_id. Logstash defaults the group\_id to `logstash` so that is why only one is doing anything. [https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html#plugins-inputs-kafka-group\_id](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html#plugins-inputs-kafka-group_id)

---

<div class="post-metadata">

**Author:** ![kirill\_polishchuk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kirill_polishchuk/32/6489_2.png) [@kirill\_polishchuk](https://discuss.elastic.co/u/kirill_polishchuk)\
**Post date:** [March 25, 2016, 4:41pm UTC](https://discuss.elastic.co/t/two-logstash-servers-with-kafka/45238/3 "2016-03-25T16:41:31Z")

</div>

Thank you, Joe, for the answer.

I have tried to create topics with multiple partitions (for example 10), and then in logstash Input, for back-end, I set `consumer_threads` to **5** on each. This solution resolved my issue.

Actually with your way, with different `group_id` I will get the same output from Kafka for each back-end.

> If all the consumer instances have different consumer groups, then this works like publish-subscribe and all messages are broadcast to all consumers  
> [Apache Kafka](http://kafka.apache.org/documentation.html#intro_consumers)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:05am UTC](https://discuss.elastic.co/t/two-logstash-servers-with-kafka/45238/4 "2017-07-06T05:05:17Z")

</div>