Relationship between pipleline workers and Kafka/ES plugin workers

Elastic Stack Logstash
1

For Logstash 2.3, what is the relationship between the number of pipeline workers (which by default is equal to the number of cores) and the following:

  • Kafka output plugin parameter "workers".

  • Elasticsearch output plugin parameter "workers".

  • Kafka input plugin parameter "consumer_threads".

The default value for all three of these parameters is 1. For maximum throughput should we change the value of "workers" parameter to be equal to number of pipeline workers and make "consumer_threads" equal to the number of partitions in the topic?

2

The 2.3 logstash-input-kafka doesn't gain anything from bumping consumer_threads. Check out https://github.com/logstash-plugins/logstash-input-kafka/pull/79 for Kafka 0.8.X or for Kafka 0.9.X look at Logstash 5 beta. Then I'd say it actually has an effect on performance.

As for keeping the number of threads consistent across the pipeline, it really depends on your setup and you'll have to test. For the Kafka input you don't want more threads on a consumer group than partitions for that topic.