Following is the log-stash configuration used for capturing the events from various nodes streaming syslog events and the events are buffered in Redis queue for processing.
input {
udp
{
port => 514
buffer_size => 32768
queue_size => 300000
workers => 10
}
}
output {
redis {
host => '10.1.0.16'
data_type => 'list'
key => 'events'
port => 6379
batch => true
batch_events => 1000
batch_timeout => 5
workers => 10
}
}
With above configuration, we have achieved 800 EPS without UDP data loss.
In our test environment, Logstash 2.0 is deployed on win8 64 bit, i7 2.6 GHz, 8 GB RAM.
Please share your thoughts to improve processing of events at 50 K events per sec without data loss.
Also suggest the ideal system hardware for such a scenario.