Redis input pipeline intermittently stops for couple minutes


One of my "heavy duty" pipelines started acting strangely in the past weeks. I get alerts that ingesting stopped then I see that after a spike everything goes back to normal. I checked Elastic and Logstash logs on all nodes and there isn't anything. The Redis server log is empty since it was started a couple of months ago.

Logstash pipeline event rate:

Redis metrics:

As you see when Logstash stops getting events, Redis memory usage spikes which means the data flow is consistent. I checked various server related metrics but I don't see anything which would indicate why this particular pipeline stops. Longer (~2 minutes) pauses happen a couple of times a day (3-8) but when I check the event rate I spotted several occasions where it dropped to 0 but then returned working. However, none of the other pipelines (where data flow is consistent) shows a similar pattern.

Specs: Logstash 7.0.0; Redis server v=5.0.4

Is there any way to debug this and solve the problem? I'm out of ideas on what to check.

Thank you!

If you have multiple pipelines in a single instance and only one of them stops processing then I would lean towards it being a blocked output and back-pressure shutting down the pipeline. You could test that by enabling a persistent queue on the output and then monitoring the pipeline stats API which will indicate how much of the queue is full. If that rules out a blocked output it suggests a problem on the input side.

Thank you!
I'll dive into the topic and post results later.