One of my "heavy duty" pipelines started acting strangely in the past weeks. I get alerts that ingesting stopped then I see that after a spike everything goes back to normal. I checked Elastic and Logstash logs on all nodes and there isn't anything. The Redis server log is empty since it was started a couple of months ago.
Logstash pipeline event rate:
As you see when Logstash stops getting events, Redis memory usage spikes which means the data flow is consistent. I checked various server related metrics but I don't see anything which would indicate why this particular pipeline stops. Longer (~2 minutes) pauses happen a couple of times a day (3-8) but when I check the event rate I spotted several occasions where it dropped to 0 but then returned working. However, none of the other pipelines (where data flow is consistent) shows a similar pattern.
Specs: Logstash 7.0.0; Redis server v=5.0.4
Is there any way to debug this and solve the problem? I'm out of ideas on what to check.