Redis output plugin "Redis::TimeoutError" never recovers

[nikhilo (Nikhil Owalekar)]

We have been trying to introduce redis in our logstash pipeline in order to spread the parsing load evenly among multiple logstash servers.
Note: Tried this approach with logstash 1.4.2 and 5.1.1

Unfortunately some of the logstash agents intermittently experienced "Redis::TimeoutError". It was happening more frequently, so I followed some suggestions found on the web, and configured redis output (added batch, increased timeout) like so,

    redis {
  host => 'logstash-redis-elasticache'
  data_type => 'list'
  key => 'logstash-%{type}'
  batch => true
  batch_events => 25 <---- cannot use more than this, redis times out more frequently
  batch_timeout => 10
  timeout => 12
  reconnect_interval => 30
}

This still has not resolved the problem. (Less frequent timeouts as compared to non-batch config)
I can understand that Redis can time out under high load, but from system metrics for the redis, it doesn't seem to be under high load at all.

Unfortunately we are running on Ubuntu 12.04, and this might not be helping the situation.
But, I would certainly hope that the output plugin should somehow succeed on one of the retries, and just resume sending the logs. Sometimes it does! But not every time. That's the problem.

Any help appreciated!

Thanks,
Nikhil