My ELK pipeline is: nxlog on Win servers (about 35-40) > one logstash server > ES
When I look at the events in ES they are hours old, but I do eventually get them- its just always a few hours behind. No errors or dropped events anywhere and CPU,Mem,Disk utilization is low on ES and logstash servers.
Looking at nxlog on the endpoints it appears to be the bottleneck. It is taking a long time to ship logs. The nxlog process however is running with practically no resource utilization.
I'm using nxlog to ship via tcp so it is using flow control. So now I suspect the logstash servers NIC is over saturated and is blocking nxlogs requests (via flow control).
My Logstash server's CPU, Mem, and iowait are low.
How can I confirm the NIC is overwhelmed? Latency from endpoints to the logstash server is relatively low and consistent (200ms windows ping).
Running centos7 on my logstash server.
Here is the output of netstat -s
Ip:
319335 total packets received
0 forwarded
0 incoming packets discarded
256063 incoming packets delivered
111065 requests sent out
40 dropped because of missing route
Icmp:
2972 ICMP messages received
11 input ICMP message failed.
ICMP input histogram:
destination unreachable: 11
echo requests: 2961
2972 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
destination unreachable: 11
echo replies: 2961
IcmpMsg:
InType3: 11
InType8: 2961
OutType0: 2961
OutType3: 11
Tcp:
1321 active connections openings
670 passive connection openings
5 failed connection attempts
22 connection resets received
69 connections established
257515 segments received
304706 segments send out
59 segments retransmited
0 bad segments received.
1265 resets sent
Udp:
1615 packets received
0 packets to unknown port received.
0 packet receive errors
1615 packets sent
0 receive buffer errors
0 send buffer errors
UdpLite:
TcpExt:
12 invalid SYN cookies received
629 TCP sockets finished time wait in fast timer
13153 delayed acks sent
Quick ack mode was activated 24 times
83163 packet headers predicted
16087 acknowledgments not containing data payload received
66796 predicted acknowledgments
2 other TCP timeouts
TCPLossProbes: 71
TCPLossProbeRecovery: 48
137 DSACKs sent for old packets
49 DSACKs received
630 connections reset due to unexpected data
1 connections aborted due to timeout
TCPDSACKIgnoredNoUndo: 48
TCPRcvCoalesce: 126430
TCPOFOQueue: 489
TCPAutoCorking: 68
TCPFromZeroWindowAdv: 5704
TCPToZeroWindowAdv: 5735
TCPWantZeroWindowAdv: 10026
TCPSynRetrans: 1
TCPOrigDataSent: 227198
TCPHystartTrainDetect: 16
TCPHystartTrainCwnd: 500
TCPHystartDelayDetect: 2
TCPHystartDelayCwnd: 125
IpExt:
InNoRoutes: 60
InBcastPkts: 1294
InOctets: 201249679
OutOctets: 314393682
InBcastOctets: 169711
InNoECTPkts: 303209
InECT0Pkts: 16126