Im sending all my beat* data into kafka and reading it from there via logstash (running on kubernetes). I use kafka for lots of other purposes and I know we can read 1M records a second using a simple consumer. When I look at the stats for my cluster it appears we're getting ~50K records / sec.
I suspect I have something misconfigured in my logstash setup. Not sure where to look to find tips/tricks for optimizing this path.
System: all 7.3.0
5 data nodes running on kubernetes hosts with 56 cores/64G RAM
5 ingest nodes
3 master nodes
file/metric beat kafka topics have 20 partitions and 20 consumers for each - also running in kubernetes.