# How many consumer\_threads of Kafka input to configure if duplicate Logstash instances

**URL:** <https://discuss.elastic.co/t/how-many-consumer-threads-of-kafka-input-to-configure-if-duplicate-logstash-instances/137402>\
**Category:** Logstash\
**Created:** [June 26, 2018, 9:13am UTC](https://discuss.elastic.co/t/how-many-consumer-threads-of-kafka-input-to-configure-if-duplicate-logstash-instances/137402 "2018-06-26T09:13:57Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![elasticheart](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/elasticheart/32/65189_2.png) [@elasticheart](https://discuss.elastic.co/u/elasticheart)\
**Post date:** [June 26, 2018, 9:13am UTC](https://discuss.elastic.co/t/how-many-consumer-threads-of-kafka-input-to-configure-if-duplicate-logstash-instances/137402/1 "2018-06-26T09:13:57Z")

</div>

Hi,

I am using ELK GA 6.3.0. I am consuming from a Kafka topic which has 11 partitions. My configuration is like below;

```
input {
	kafka{
		group_id => "group1"
		topics => ["topic1"]
		bootstrap_servers => "192.168.0.1:9092,192.168.0.2:9092,192.168.0.3:9092"
		codec => "json"
	}
}

```

In the **[plugin doc](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html#plugins-inputs-kafka-consumer_threads)**, it is mentioned that;

> Ideally you should have as many threads as the number of partitions for a perfect balance — more threads than partitions means that some threads will be idle

So I changed my configuration like this;

```
input {
	kafka{
		group_id => "group1"
		topics => ["topic1"]
		bootstrap_servers => "192.168.0.1:9092,192.168.0.2:9092,192.168.0.3:9092"
		codec => "json"
		consumer_threads => "11"
	}
}

```

I can understand this if I have one instance of Logstash. But I am running 2 Logstash instances. My intentions are;

1. Consume from topic even if one Logstash terminates / fails
2. Load distribution.

So my question is, how many `consumer_threads` should I give - is it **11 for both** or **5 for one and 6 for other**?

Thanks.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 24, 2018, 9:18am UTC](https://discuss.elastic.co/t/how-many-consumer-threads-of-kafka-input-to-configure-if-duplicate-logstash-instances/137402/2 "2018-07-24T09:18:07Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.