TLDR: It appears Logstash's emission rate is limited in some way other than provided resources.
Started having a weird issue yesterday with Logstash performing very poorly. Typical ingest over the past couple weeks has been around 4,500 events/second. Logstash has been keeping up fine with that but suddenly, the disk queue started to fill yesterday even though there wasn't any kind of unusual increase in events. Once the queue hit about 25GB (Max capacity 100GB), I started shutting down agents to reduce ingest rate. I noticed that as the ingest rate went down, so too did the emission rate. It wasn't at a 1:1 ratio but it was still lowering near the same amount as ingest. After the queue emptied out I turned all the agents back on and the queue began to fill again but eventually tapered off around 12GB by the time I left for the day. I came in this morning to find that the queue never fully cleared out, even though our overnight ingest rate drops to about 1,000/sec. I've seen this thing hit 10k/sec at times so why is it failing to keep up?
It seems like if it can handle ingesting 4,500/sec and emitting 4,500/sec then it's capable of processing 9,000/sec. That appears to be wrong but why or is there something wrong with my setup?