We are using Packetbeat to capture traffic comming on a network interface card that is 1 GBPS and stack it on the Elasticsearch database. The problem we are currently trouble shouting is that as the speed of the network increases above 3.5 MB/s. Packetbeat start dropping packets from 20 to 30 percent against each IP within our Local Area Network(LAN). On the other hand, if the speed is less there are no losses.
We have tested the server network interface card (NIC), and data on the server's virtual port using tcpdump and found no losses on NIC. But when we test our Packetbeat captured packets it was always found less than the actual packets downloaded for test purpose.
My Packetbeat configuration are as follows:
packetbeat.interfaces.device: ${INTERFACE}
packetbeat.interfaces.type: af_packet
packetbeat.interfaces.buffer_size_mb: 1500
timeout: 30s
period: 10s
A snippet of logs is here:
[monitoring] log/log.go:145 Non-zero metrics in the last 30s
{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":38740,"time":{"ms":174}},"total":{"ticks":137120,"time":{"ms":760},"value":137120},"user":{"ticks":98380,"time":{"ms":586}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":7},"info":{"ephemeral_id":"15964248-a18e-4c31-96ad-173afe7cdc98","uptime":{"ms":8250084}},"memstats":{"gc_next":39295152,"memory_alloc":30576608,"memory_total":4444545480},"runtime":{"goroutines":38}},"dns":{"unmatched_responses":28},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":472,"batches":18,"total":472},"read":{"bytes":9037},"write":{"bytes":571594}},"pipeline":{"clients":14,"events":{"active":0,"published":472,"total":472},"queue":{"acked":472}}},"system":{"load":{"1":1.37,"15":1.14,"5":1.05,"norm":{"1":0.3425,"15":0.285,"5":0.2625}}},"tcp":{"dropped_because_of_gaps":2}}}}
Our server hardware specs are as follows:
- 16 VCPU
- 54 GB RAM
- 400 HD
- 1 GBPS Network Interface Card (NIC)