Logstash pooling data from kafka much faster with multiple processes

Elastic Stack Logstash
1

I run logstash like this

/opt/logstash/bin/logstash agent -f /opt/logstash/config/logstash.conf -w 48

this is config

input {
kafka {
type => "nginx"
zk_connect => "zk1:2181,zk2:2181,zk3:2181"
codec => "plain"
topic_id => "api_events"
consumer_threads => 16
queue_size => 10000
rebalance_backoff_ms => 10000
rebalance_max_retries => 10
}
}

output {
if [type] == "nginx" {
elasticsearch {
index => "nginx-%{+YYYY.MM.dd}"
hosts => ["es1","es2","es3"]
flush_size => 100000
}
}
}

**Whit this config it process 100K doc in 30 sec **

But when im running 16 processes with same config but consumer_threads => 1 it runs much faster (500K in 30 sec)

I wont to run just one process with multaple forks and process same rate of documents.
Did anyone have this problem or know how fix this ?

Thanks in advance

2

One thing that changes when you run 16 processes instead of 1 is the number of connections to Elasticsearch. Have you tried increasing the number of workers in the elasticsearch output to 16 or 48 when running just a single process? Does that make a difference?

3

Thanks this solved my problem