Hi there, we are currently trying to use Logstash and Filebeat to process a large netflow stream. We are running filebeat and logstash in docker containers using the official elastic search container images. The server running the netflow collector has 16GB RAM and 16 cores as well as a very fast network connection.
We are sending about 7k netflow packets per second to this netflow collector.
Logstash is using about 50% (6GB) of the Java Heap (10GB RAM) and about 25% CPU (4 cores). The monitoring for Logstash shows that it is only processing 2300 events per second. I don't see any event drops, input = output.
Filebeat is using much less memory 300MB, but uses way more CPU (50% or 8 cores). But for some reason we only see 300 events being processes per second.
Any ideas what might be happening here? I don't think that logstash is performance constrained, but I'm still unsure why we are seeing much less packets per second.
I've done some interface pps counts on the interface itself (only receiving netflow) and I'm getting about 12k pps. This is both using cat /sys/class/net/$1/statistics/rx_packets stats and tcpdump.
root@nfl-aer-001:~# ./pps.sh ens160
TX ens160: 75 pkts/s RX ens160: 9912 pkts/s
TX ens160: 87 pkts/s RX ens160: 11041 pkts/s
TX ens160: 69 pkts/s RX ens160: 11203 pkts/s
TX ens160: 66 pkts/s RX ens160: 10974 pkts/s
root@nfl-aer-001:~# timeout 10 tcpdump -B 919400 -s 64 -n -p -i ens160 udp port 2055 -w testfile
tcpdump: listening on ens160, link-type EN10MB (Ethernet), capture size 64 bytes
112726 packets captured
117234 packets received by filter
2714 packets dropped by kernel