Logstash forwarder can not keep up with traffic

Logstash forwarder can not keep up with 10TPS traffic. After running traffic over the weekend at 19TPS split over 2 Core VMs, the forwarder took several hours to catch up (send the events queued). I suspect it to be a configuration issue since we are only sending 1024 events per second/run via the forwarder. Maybe we can change that to send more events per second/run. I stopped traffic at 9:49. The /var/log/logstash-forwarder/logstash-forwarder.err
shows that it is still sending data at 12:17 and has not stopped yet. We are using Logstash version 5.1.2. Kindly assist

[root@CORE1 ~]# tail /var/log/logstash-forwarder/logstash-forwarder.err
2017/02/13 12:16:56.834572 Registrar: processing 1024 events
2017/02/13 12:16:58.229106 Registrar: processing 1024 events
2017/02/13 12:16:59.361461 Registrar: processing 1024 events
2017/02/13 12:17:00.781646 Registrar: processing 1024 events
2017/02/13 12:17:02.233247 Registrar: processing 1024 events
2017/02/13 12:17:03.735394 Registrar: processing 1024 events
2017/02/13 12:17:05.434871 Registrar: processing 1024 events
2017/02/13 12:17:06.716610 Registrar: processing 1024 events
2017/02/13 12:17:08.392964 Registrar: processing 1024 events
2017/02/13 12:17:10.113604 Registrar: processing 1024 events
[root@CORE1 ~]# tail /var/log/logstash-forwarder/logstash-forwarder.err
2017/02/13 12:17:00.781646 Registrar: processing 1024 events
2017/02/13 12:17:02.233247 Registrar: processing 1024 events
2017/02/13 12:17:03.735394 Registrar: processing 1024 events
2017/02/13 12:17:05.434871 Registrar: processing 1024 events
2017/02/13 12:17:06.716610 Registrar: processing 1024 events
2017/02/13 12:17:08.392964 Registrar: processing 1024 events
2017/02/13 12:17:10.113604 Registrar: processing 1024 events
2017/02/13 12:17:11.550328 Registrar: processing 1024 events
2017/02/13 12:17:13.108394 Registrar: processing 1024 events
2017/02/13 12:17:14.706517 Registrar: processing 1024 events
[root@CORE1 ~]#

It finally caught up at 18:26

[root@CORE1 ~]# tail /var/log/logstash-forwarder/logstash-forwarder.err
2017/02/13 18:26:35.648608 Registrar: processing 1024 events
2017/02/13 18:26:37.205799 Registrar: processing 1024 events
2017/02/13 18:26:38.710270 Registrar: processing 1024 events
2017/02/13 18:26:40.189578 Registrar: processing 1024 events
2017/02/13 18:26:41.930550 Registrar: processing 1024 events
2017/02/13 18:26:43.474398 Registrar: processing 1024 events
2017/02/13 18:26:44.985647 Registrar: processing 1024 events
2017/02/13 18:26:46.549380 Registrar: processing 1024 events
2017/02/13 18:26:48.059527 Registrar: processing 1024 events
2017/02/13 18:26:50.824500 Registrar: processing 104 events
[root@CORE1 ~]#

Why are you still using logstash-forwarder? This was deprecated and replaced by Filebeat a long time ago.

Hi Christian ,This is being used in my project .I have got ticket to solve this issue. Since I am new to logstash I need some pointers which could help me to fix this issue.

I would recommend you switch to using Filebeat. Logstash-forwarder has not been maintained for over 3 years, and I have not used it since Filebeat was introduced.

Logstash-forwarder can only send events as fast as the receiving system can accept them. How do you know it is not Logstash or Elasticsearch that is limiting the throughput?

I am not sure though. I started reading about Logstash-forwarder today only .I guess i have to read ElasticSearch and know about how both are inter-related:). But is there any pointer you would suggest like location of logstash config etc .May be the configuration in logstash doesnot allow logs to be consumed

Where are you getting the 10 and 19 TPS from? It looks like Logstash-forwarder is sending hundreds of events per second.

The figure was given by performance test team.

Then I would recommend you talk to them to clarify where it comes from. Do you have any monitoring installed?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.