# Relationship between pipleline workers and Kafka/ES plugin workers

**URL:** https://discuss.elastic.co/t/relationship-between-pipleline-workers-and-kafka-es-plugin-workers/46608
**Category:** Logstash
**Created:** [April 6, 2016, 9:30pm UTC](https://discuss.elastic.co/t/relationship-between-pipleline-workers-and-kafka-es-plugin-workers/46608 "2016-04-06T21:30:51Z")
**Posts on this page:** 3
**Page:** 1

<div class="post-metadata">

### Author: ![dr\_work](https://avatars.discourse-cdn.com/v4/letter/d/4af34b/32.png) [@dr\_work](https://discuss.elastic.co/u/dr_work)
#### Post date: [April 6, 2016, 9:30pm UTC](https://discuss.elastic.co/t/relationship-between-pipleline-workers-and-kafka-es-plugin-workers/46608/1 "2016-04-06T21:30:51Z")

</div>

For Logstash 2.3, what is the relationship between the number of pipeline workers (which by default is equal to the number of cores) and the following:

- **Kafka output plugin** parameter **"workers"**.

- **Elasticsearch output plugin** parameter **"workers"**.

- **Kafka input plugin** parameter **"consumer\_threads"**.

The default value for all three of these parameters is 1. For maximum throughput should we change the value of "workers" parameter to be equal to number of pipeline workers and make "consumer\_threads" equal to the number of partitions in the topic?

---

<div class="post-metadata">

### Author: ![Joe\_Lawson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joe_lawson/32/3390_2.png) [@Joe\_Lawson](https://discuss.elastic.co/u/Joe_Lawson)
#### Post date: [April 13, 2016, 1:21am UTC](https://discuss.elastic.co/t/relationship-between-pipleline-workers-and-kafka-es-plugin-workers/46608/2 "2016-04-13T01:21:29Z")

</div>

The 2.3 logstash-input-kafka doesn't gain anything from bumping consumer\_threads. Check out [https://github.com/logstash-plugins/logstash-input-kafka/pull/79](https://github.com/logstash-plugins/logstash-input-kafka/pull/79) for Kafka 0.8.X or for Kafka 0.9.X look at Logstash 5 beta. Then I'd say it actually has an effect on performance.

As for keeping the number of threads consistent across the pipeline, it really depends on your setup and you'll have to test. For the Kafka input you don't want more threads on a consumer group than partitions for that topic.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 5:02am UTC](https://discuss.elastic.co/t/relationship-between-pipleline-workers-and-kafka-es-plugin-workers/46608/3 "2017-07-06T05:02:39Z")

</div>


