Hi, I'm suffering a serious tuning issue processing messages from a kafka queue.
The architecture is the following:
ASA Firewall -> logstash input -> kafka queue (32 partitions) -> logstash output (4 workers) -> elastic search
On a 4 vCpu 32gb vmware virtual machine (CentOS 7 OS).
The firewall produces an average flow of 3800 mess/s that have to be stored, read, PARSED and loaded into ES, and the performance issue is obiviously related to parsing with 14 matches to try for each message.
I've tried with anchors, splitting and any other performance tuning but the average output is "only" 4200 messages/s and the logstash process is using about 270% cpu instead of a full 400% (even on a 8 cores testing vm ).
I have to reach an average output of 6000 messages/s to empty the kafka queue in case of emergency.