How to limit the http output from logstash?


(Santosh Gupta) #1

I have a pipeline which uses kafka as i/p and HTTP as o/p.
I want to limit the HTTP requests that are sent from logstash o/p.
Is there a way to do that?


(Magnus Bäck) #2

What is the desired behavior if more events arrive to the Kafka topic than the allowed Logstash output rate? Queue up the events in Kafka for later processing or drop overflow events in Logstash?


(Santosh Gupta) #3

I don't want to drop any event. I just want to limit the output rate, so that when bursts of input comes, they don't overload my HTTP servers.


(Magnus Bäck) #4

Perhaps the throttle filter in combination with a sleep filter will do?


(Santosh Gupta) #5

Would using "queue.max_events" solve the issue?
As per the documentation -
"When the queue is full, Logstash puts back pressure on the inputs to stall data flowing into Logstash. This mechanism helps Logstash control the rate of data flow at the input stage without overwhelming outputs like Elasticsearch."

Ref. - https://www.elastic.co/guide/en/logstash/current/persistent-queues.html


(Magnus Bäck) #6

queue.max_events doesn't provide general rate limiting.


(system) #7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.