There seems to be an issue with the redis input with logstash. From what I can tell, it is processing the redis lists but it is going slowly. The length of the lists keep growing.
I haven't able to isolate the problem.
Any ideas?
Thanks,
Jared
On a side note, I don't see anything interesting in the log file.
What outputs do you have? Since Logstash doesn't have any internal buffers to speak of, the input rate will be equal to the output rate. What's your current message rate? What appears to be the bottleneck; CPU, memory, I/O, or network?
It looks like the outputs are the cause of it; I remove all outputs except the one to ElasticSearch. However, I can not explain why it was happening. The other outputs are to file, pagerduty, email, riemann, and slack. I have conditional tags that determine which output to used. Besides file output and ElasticSearch output, the outputs are used only when the message is tagged critical. It was working fine for months. The only changed that I made was add riemann because Slack output was working. But, commented out riemann output didn't solve the issue.