I am using logstash-forwarder to send nginx access.log with flow:
access.log -> logstash-forwarder 0.4.0-> logstash 1.5.4 -> redis 3.0.1
The logstash only run for storing data to redis. Due to high activity on nginx, data on access.log increase rapidly and unfortunately logstash-forwarder can't catch up the log. Logstash-forwarder is running with params "-spool-size 32768 -log-to-syslog -harvest-buffer-size 1024". From syslog, processing rate is 32k events/10second (3.2K/second)
====
Feb 26 04:13:16 host1 logstash-forwarder[20374]: 2016/02/26 16:13:16.586663 Registrar: processing 32768 events
Feb 26 04:13:26 host1 logstash-forwarder[20374]: 2016/02/26 16:13:26.480681 Registrar: processing 32768 events
Feb 26 04:13:36 host1 logstash-forwarder[20374]: 2016/02/26 16:13:36.707035 Registrar: processing 32768 events
Here, the bottle neck is logstash-forwarder beacuse I don't see any bottle neck on network, disk & CPU in servers. Anyone has same issue?
Please your advice