# Logstash pooling data from kafka much faster with multiple processes
**URL:** https://discuss.elastic.co/t/logstash-pooling-data-from-kafka-much-faster-with-multiple-processes/62014
**Category:** Logstash
**Created:** [October 2, 2016, 8:18pm UTC](https://discuss.elastic.co/t/logstash-pooling-data-from-kafka-much-faster-with-multiple-processes/62014 "2016-10-02T20:18:38Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@rpuserh](https://discuss.elastic.co/u/rpuserh)
#### Post date: [October 2, 2016, 8:18pm UTC](https://discuss.elastic.co/t/logstash-pooling-data-from-kafka-much-faster-with-multiple-processes/62014/1 "2016-10-02T20:18:38Z")
**I run logstash like this**
_/opt/logstash/bin/logstash agent -f /opt/logstash/config/logstash.conf -w 48_
**this is config**
input {
kafka {
type =\> "nginx"
zk\_connect =\> "zk1:2181,zk2:2181,zk3:2181"
codec =\> "plain"
topic\_id =\> "api\_events"
consumer\_threads =\> 16
queue\_size =\> 10000
rebalance\_backoff\_ms =\> 10000
rebalance\_max\_retries =\> 10
}
}
output {
if [type] == "nginx" {
elasticsearch {
index =\> "nginx-%{+YYYY.MM.dd}"
hosts =\> ["es1","es2","es3"]
flush\_size =\> 100000
}
}
}
\*\*Whit this config it process 100K doc in 30 sec \*\*
**But when im running 16 processes with same config but consumer\_threads =\> 1 it runs much faster (500K in 30 sec)**
**I wont to run just one process with multaple forks and process same rate of documents.**
**Did anyone have this problem or know how fix this ?**
Thanks in advance
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [October 3, 2016, 3:06am UTC](https://discuss.elastic.co/t/logstash-pooling-data-from-kafka-much-faster-with-multiple-processes/62014/2 "2016-10-03T03:06:50Z")
One thing that changes when you run 16 processes instead of 1 is the number of connections to Elasticsearch. Have you tried increasing the number of [workers](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-workers) in the elasticsearch output to 16 or 48 when running just a single process? Does that make a difference?
---
### Author:  [@rpuserh](https://discuss.elastic.co/u/rpuserh)
#### Post date: [October 3, 2016, 4:50pm UTC](https://discuss.elastic.co/t/logstash-pooling-data-from-kafka-much-faster-with-multiple-processes/62014/3 "2016-10-03T16:50:10Z")
Thanks this solved my problem
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 4:36am UTC](https://discuss.elastic.co/t/logstash-pooling-data-from-kafka-much-faster-with-multiple-processes/62014/4 "2017-07-06T04:36:03Z")