# Multiple logstash instances consume kafka, only one is working

**URL:** <https://discuss.elastic.co/t/multiple-logstash-instances-consume-kafka-only-one-is-working/321880>\
**Category:** Logstash\
**Created:** [December 23, 2022, 5:29am UTC](https://discuss.elastic.co/t/multiple-logstash-instances-consume-kafka-only-one-is-working/321880 "2022-12-23T05:29:06Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![TemplateXu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/templatexu/32/115143_2.png) [@TemplateXu](https://discuss.elastic.co/u/TemplateXu)\
**Post date:** [December 23, 2022, 5:29am UTC](https://discuss.elastic.co/t/multiple-logstash-instances-consume-kafka-only-one-is-working/321880/1 "2022-12-23T05:29:06Z")

</div>

Multiple logstash instances consume log messages in kafka, but only one of them is working. When I close one of them, the other starts to work. How to make two logstash instances work together

---

<div class="post-metadata">

**Author:** ![hendry.lim](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendry.lim/32/71328_2.png) [@hendry.lim](https://discuss.elastic.co/u/hendry.lim)\
**Post date:** [December 23, 2022, 6:45am UTC](https://discuss.elastic.co/t/multiple-logstash-instances-consume-kafka-only-one-is-working/321880/2 "2022-12-23T06:45:47Z")

</div>

> [@TemplateXu](#):
>
> but only one of them is working

You meant only one Logstash instance is actively consuming from Kafka topic?  
How many topics and partitions do you have?  
How many consumer threads have you configured in `kafka` input?

If you have 1 topic and 2 partitions, you should set `consumer_threads` in each Logstash instance to be `1`. This will allow both Logstash instances to consume from the same topic concurrently from different partitions.  
i.e. the total number of `consumer_threads` across your Logstash instances (in the same client group) should be equal to the number of partitions you are consuming from.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 20, 2023, 6:45am UTC](https://discuss.elastic.co/t/multiple-logstash-instances-consume-kafka-only-one-is-working/321880/3 "2023-01-20T06:45:54Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.