I've tried to look up every available documentation, example, blog post that's available on the net but I couldn't find anything useful on how to speed up ingesting from Kafka. Logstash barely uses any resources (10% of 16vCPU, 1GB RAM), same for Kafka. Nodes are set up in the same data center so network isn't a bottleneck either. The only thing that changed the throughput was the "consumer_threads" setting which is set to add up to the total number of partitions (15 per LS instance). When I changed anything other (e.g., fetch_max_bytes => 157286400) nothing happened, so I removed those leaving at the default setting.
I did read similar issues from the past but those were mostly unanswered. Unforunately, the partitions show significant lag (offset -X million) and I can't work out how to solve this since clearly throwing resources at the problem isn't a solution. I also couldn't figure out if I set something in the Kafka input configuration, would that change the limits on the Kafka server? Naturally, I elevated the limits on the Kafka servers as well, I was just wondering.
What am I missing? How can I squeeze out more (a lot) throughput from Logstash?
Logstash version: 7.9.2
Kafka version: 2.3.0