Workers option logstash input

Elastic Stack Logstash
1

Does the option "workers" works with redis input????

input {
redis {
host => "172.16.122.120"
data_type => "list"
type => "redis-input"
key => "logstash"
workers => 8
}
}

2

The docs say.......No!

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-redis.html

3

appreciate it man :slightly_smiling:
many thanks

4

If you are wanting it to work faster, something you could try is to change the batch_count. From what I understand, by default it pulls off one event at a time. Changing the batch count to 10 or 50 will increase how much it pulls at once. In our case it made it work significantly faster. I can't remember the specifics anymore, but I think it changed one logstash node to be able to handle around 3k eps instead of 2k eps. Try a few different values to see where you start getting diminishing returns.
It only really helps when there are a lot of events stacked up on Redis though. If the Redis queue is less than your batch_count you won't see much benefit.

1 Like
5

Thanks man, seems very helpful, i'll give it a try

6

also you can try use more instances of logstash with the same config.

7

with input redis, the option is named «threads».
https://www.elastic.co/guide/en/logstash/current/plugins-inputs-redis.html#plugins-inputs-redis-threads

8

added, thanks

9

Also, I have written a question about tuning this redis parameter with the new NG-Pipeline as of 2.2.

It may be of interest to you.

Good luck
Bruno Lavoie