I have a pipeline which uses kafka as i/p and HTTP as o/p.
I want to limit the HTTP requests that are sent from logstash o/p.
Is there a way to do that?
What is the desired behavior if more events arrive to the Kafka topic than the allowed Logstash output rate? Queue up the events in Kafka for later processing or drop overflow events in Logstash?
I don't want to drop any event. I just want to limit the output rate, so that when bursts of input comes, they don't overload my HTTP servers.
Perhaps the throttle filter in combination with a sleep filter will do?
Would using "queue.max_events" solve the issue?
As per the documentation -
"When the queue is full, Logstash puts back pressure on the inputs to stall data flowing into Logstash. This mechanism helps Logstash control the rate of data flow at the input stage without overwhelming outputs like Elasticsearch."
Ref. - https://www.elastic.co/guide/en/logstash/current/persistent-queues.html
queue.max_events doesn't provide general rate limiting.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.