REDIS input stops working?

Elastic Stack Logstash
1

Hello,
We currently have a server with redis installed, and several logstash instances. Half of these instances listen on a port and output to redis, and half of them input from redis, filter, and output to elasticsearch. I have noticed that after a few days, the input stops working. Using redis-stat, I see that the redis db size keeps increasing, because the logs aren't being 'blpoped'. This is then causing the server to slow to a crawl because of memory usage, and then cluster instability inevitably results. I'm having trouble finding the cause of this error. The logs don't show anything, the .log files just show some java.lang.IllegalArgumentException: Invalid format error for the timestamp, but those go back several days. The redis input instances stopped working sometime between 4 PM yesterday and 10 AM today. Also, the last timestamp for the logs is Aug 20, at 3:35 AM. I know for a fact it was working most of the day yesterday. Where should I be looking? I see nothing out of the ordinary in the redis logs as well. There are some memory errors in the elasticsearch logs, but I'm not sure if that's part of the issue. If logstash is not able to output to elasticsearch, will it stop input as well?

EDIT:
I should note, I can get it working again by killing the logstash processes and restarting them. Then it seems to work fine. Also, the output to redis (different logstash instances) work fine the entire time.

2

I see this as well. I have 6 logstash servers, each with their own redis instance. (All 6 logstash processes pull from all 6 redis instances.)

During the day, I have occasional redit input plugin problems. (I see it from monitoring which notices the redis db for an instance growing abnormally.)

If I manually restart logstash, things return back to normal. However, if I just let it sit there, eventually it fixes itself. Maybe with an hour or two.

I've seen this behavior for many releases now. I haven't been able to spend time on tracking it down.

3

What version of logstash are you running? It may be related to this on github #3361 and the fix here #3378

I am going to try and upgrade from 1.5.0 to 1.5.1 (tried to jump directly to 1.5.4 but our production systems all run java 7, which is deprecated on that version)

4

I'm running 1.5.1.