hi,in our network we use packetbeat with 'af_packet' mode on readhat 6.5 (22 core CPU, 768G memory, 16 SATA disks, 10Gbs NIC) to analyze HTTP traffic which is about 4Gbps, and we find that it does not work very well, too much packets were missed. so I want to know the performance limit of packetbeat? and is there some advice from elastic for us to speedup performance of packetbeat ?
btw, I know there are some high performance solutions for packet traffic on Linux such as DPDK and so on, will packetbeat consider them? Thanks.
Actually af_packet can be quite fast, as af_packet supports multiple receive queues, allowing for some parallelism. DPDK (and others) performance gains are mostly due to bypassing most of the network layer + (somewhat more important) parallelising processing, by having multiple receive queues (be it virtual or physical). Parallelising processing is about load-balancing. But for analysing a stable hash based load-balancing is required -> load distribution on queues and analysers do depend on actual traffic patterns.
Unfortunately packetbeat currently does not support multiple af_packet receive queues, but configures only one queue. Therefore all analysis is fully single threaded. The only way to get some load-balancing is to have multiple devices (e.g. either your network adapter+driver supports the creation of virtual devices to sniff from, or your tap support some load balancing) and start one packetbeat per devices.
Another potential problem is with outputs. If your target system is not able to cope with the number of events packetbeat generates, transactions are dropped by packetbeat (giving you some kind of sampling).
Thanks for reply. so can I start mutiple packetbeat instances with different BPFs but on only one device to supporting load balancing? I have 768G memory and 22 CPU cores, I want to find a way to improve efficiency. Thanks
All these packetbeat instances would still server the same queue. Normally load-balancing should be done with multiple receive queue, with help of the NIC. Just using BPF, the filter still must be applied on every single packet + you will have a multitude of filters to be executed, before a packet hits packetbeat. I don't think you will see much benefit by running multiple packetbeat with multiple bpf filters.
The http parser failed on some packets. Parsing errors could be triggered by packet-loss, or when packetbeat is first started up. For parsing the application layer, the parsers in packetbeat must be in sync with the actual TCP streams. On startup or after packetloss, packetbeat is not in sync -> starts parsing somewhere in the middle of a http message. When hitting parsing errors, packetbeat will throw away the TCP connections current start and attempt to parse again, until packetbeat is eventually in-sync.
Alternatively to packetbeat not being in sync with the network traffic, this could also be a bug. Anyways, having a stack-trace indicates the parse could be made more robust on errors.
Can you please post the full stack-trace after this message occurs? Also check your logs for messages indicating packet drops (gaps).
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.