Filebeat netflow events per second - losing data?

[sgreszcz (Stephen Greszczyszyn)]

Hi there, we are currently trying to use Logstash and Filebeat to process a large netflow stream. We are running filebeat and logstash in docker containers using the official elastic search container images. The server running the netflow collector has 16GB RAM and 16 cores as well as a very fast network connection.

We are sending about 7k netflow packets per second to this netflow collector.

Filebeat is using much less memory 300MB, but uses way more CPU (50% or 8 cores). But for some reason we only see 300 events being processes per second.

Any ideas what might be happening here? I don't think that filebeat is performance constrained, but I'm still unsure why we are seeing much less packets per second.

I've done some interface pps counts on the interface itself (only receiving netflow) and I'm getting about 12k pps. This is both using cat /sys/class/net/$1/statistics/rx_packets stats and tcpdump.

root@nfl-aer-001:~# ./pps.sh ens160

TX ens160: 75 pkts/s RX ens160: 9912 pkts/s

TX ens160: 87 pkts/s RX ens160: 11041 pkts/s

TX ens160: 69 pkts/s RX ens160: 11203 pkts/s

TX ens160: 66 pkts/s RX ens160: 10974 pkts/s

root@nfl-aer-001:~# timeout 10 tcpdump -B 919400 -s 64 -n -p -i ens160 udp port 2055 -w testfile

tcpdump: listening on ens160, link-type EN10MB (Ethernet), capture size 64 bytes

112726 packets captured

117234 packets received by filter

2714 packets dropped by kernel

[andrewkroh]

One thing you could try is to run a quick test with filebeat outputting to /dev/null and checking it's event processing rate. Something like

systemctl stop filebeat.service
sudo filebeat -e -E output.elasticsearch.enabled=false -E output.console.enabled=true > /dev/null

When you stop the process it will log some metrics about the number of events and total uptime so you should be able to get an average rate.