Some packets are not getting captured on heavy load

Elastic Stack Beats
1

I am load testing packetbeat with 2 nginx servers at load of 2000 requests per second per server. And I see that only half were getting captured. When I tried with logging I can see these plenty of similar warning logs

2016/06/10 14:16:08.979317 http.go:421: WARN Response from unknown transaction. Ingoring.
2016/06/10 14:16:08.979368 http.go:421: WARN Response from unknown transaction. Ingoring.
2016/06/10 14:16:09.040683 tcp.go:147: WARN Gap in tcp stream. last_seq: 1225473772, seq: 1225474690, gap: 918
2016/06/10 14:16:09.040868 tcp.go:147: WARN Gap in tcp stream. last_seq: 3472751963, seq: 3472757913, gap: 5950
2016/06/10 14:16:09.041198 tcp.go:147: WARN Gap in tcp stream. last_seq: 2813628281, seq: 2813628434, gap: 153

Note: I am not using any tls.
Can anyone please help?

2

This is due to packet-loss. Packetbeat trying to passively read packets from network layer. You running packetbeat in same host as nginx servers or separately?

Some kind of load-balancing (multiple devices to read from by splitting traffic) might help.

Which outputs have you configured? You can test with console output and forward all output to /dev/null. Packetbeat still having problem with packet-loss?

3

@steffens
I am running nginx inside docker containers. And I am running packetbeat on physical machine docker0 interface which is used by docker containers.

I am using redis output which is on other machine, but both packetbeat machine and redis machine are connected by a switch.

i ran ifconfig command and I see that there were no drops or packet errors. Can you please elaborate little bit on load balancing, can we use this kind of technique for single interface?

And will console output give any other stats? , I thought it will be difficult to observe console output with this scale.

Please let me know if you need any other information.

4

idea with console output is to enable console output and send it to /dev/null .

$ packetbeat >/dev/null

The logs are still written. So you can tell if packet-loss still occurs. I don't think you can tell packet-loss from ifconfig. As services act via TCP, the services see all traffic. Only packetbeat might miss packets, due to it not being able to keep up with traffic.

5

This topic was automatically closed after 21 days. New replies are no longer allowed.