Logstash forwarder can not keep up with traffic

Elastic Stack Logstash
1

Logstash forwarder can not keep up with 10TPS traffic. After running traffic over the weekend at 19TPS split over 2 Core VMs, the forwarder took several hours to catch up (send the events queued). I suspect it to be a configuration issue since we are only sending 1024 events per second/run via the forwarder. Maybe we can change that to send more events per second/run. I stopped traffic at 9:49. The /var/log/logstash-forwarder/logstash-forwarder.err
shows that it is still sending data at 12:17 and has not stopped yet. We are using Logstash version 5.1.2. Kindly assist

[root@CORE1 ~]# tail /var/log/logstash-forwarder/logstash-forwarder.err
2017/02/13 12:16:56.834572 Registrar: processing 1024 events
2017/02/13 12:16:58.229106 Registrar: processing 1024 events
2017/02/13 12:16:59.361461 Registrar: processing 1024 events
2017/02/13 12:17:00.781646 Registrar: processing 1024 events
2017/02/13 12:17:02.233247 Registrar: processing 1024 events
2017/02/13 12:17:03.735394 Registrar: processing 1024 events
2017/02/13 12:17:05.434871 Registrar: processing 1024 events
2017/02/13 12:17:06.716610 Registrar: processing 1024 events
2017/02/13 12:17:08.392964 Registrar: processing 1024 events
2017/02/13 12:17:10.113604 Registrar: processing 1024 events
[root@CORE1 ~]# tail /var/log/logstash-forwarder/logstash-forwarder.err
2017/02/13 12:17:00.781646 Registrar: processing 1024 events
2017/02/13 12:17:02.233247 Registrar: processing 1024 events
2017/02/13 12:17:03.735394 Registrar: processing 1024 events
2017/02/13 12:17:05.434871 Registrar: processing 1024 events
2017/02/13 12:17:06.716610 Registrar: processing 1024 events
2017/02/13 12:17:08.392964 Registrar: processing 1024 events
2017/02/13 12:17:10.113604 Registrar: processing 1024 events
2017/02/13 12:17:11.550328 Registrar: processing 1024 events
2017/02/13 12:17:13.108394 Registrar: processing 1024 events
2017/02/13 12:17:14.706517 Registrar: processing 1024 events
[root@CORE1 ~]#

It finally caught up at 18:26

[root@CORE1 ~]# tail /var/log/logstash-forwarder/logstash-forwarder.err
2017/02/13 18:26:35.648608 Registrar: processing 1024 events
2017/02/13 18:26:37.205799 Registrar: processing 1024 events
2017/02/13 18:26:38.710270 Registrar: processing 1024 events
2017/02/13 18:26:40.189578 Registrar: processing 1024 events
2017/02/13 18:26:41.930550 Registrar: processing 1024 events
2017/02/13 18:26:43.474398 Registrar: processing 1024 events
2017/02/13 18:26:44.985647 Registrar: processing 1024 events
2017/02/13 18:26:46.549380 Registrar: processing 1024 events
2017/02/13 18:26:48.059527 Registrar: processing 1024 events
2017/02/13 18:26:50.824500 Registrar: processing 104 events
[root@CORE1 ~]#

2

Why are you still using logstash-forwarder? This was deprecated and replaced by Filebeat a long time ago.

3

Hi Christian ,This is being used in my project .I have got ticket to solve this issue. Since I am new to logstash I need some pointers which could help me to fix this issue.

4

I would recommend you switch to using Filebeat. Logstash-forwarder has not been maintained for over 3 years, and I have not used it since Filebeat was introduced.

Logstash-forwarder can only send events as fast as the receiving system can accept them. How do you know it is not Logstash or Elasticsearch that is limiting the throughput?

5

I am not sure though. I started reading about Logstash-forwarder today only .I guess i have to read Elasticsearch and know about how both are inter-related:). But is there any pointer you would suggest like location of logstash config etc .May be the configuration in logstash doesnot allow logs to be consumed

6

Where are you getting the 10 and 19 TPS from? It looks like Logstash-forwarder is sending hundreds of events per second.

7

The figure was given by performance test team.

8

Then I would recommend you talk to them to clarify where it comes from. Do you have any monitoring installed?

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.