Work with Redis in Logstash 1.5 and 5.1

Elastic Stack Logstash
1

I have curent logging flow:
Filebeat -> Redis -> Logstash -> Elasticsearch

At first time when I used Logstash 5.1 and Redis I found strange behavior in Logstash messages collecting from Redis. The message queue is jumping up and down randomly with trend to growing and Logstash doesn't have much time to collect messages. In the end Redis cache is flushed and huge amount of logs are lost. I tried to play with Redis input plugin in Logstash config. I've changed amount of threads, batch count and amount of workers for pipeline. The result was the same - 30%-50% CPU utilization and Redis is overflowed. I have no clue what's happening.
But. If I use Logstash 1.5 with same configuration the message queue in Redis still about zero and doesn't overflow anymore.
Can anybody explain me this strange behavior?

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.