# How to limit the http output from logstash?

**URL:** <https://discuss.elastic.co/t/how-to-limit-the-http-output-from-logstash/130099>\
**Category:** Logstash\
**Created:** [May 1, 2018, 8:16am UTC](https://discuss.elastic.co/t/how-to-limit-the-http-output-from-logstash/130099 "2018-05-01T08:16:59Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![santoshgupta](https://avatars.discourse-cdn.com/v4/letter/s/2bfe46/32.png) [@santoshgupta](https://discuss.elastic.co/u/santoshgupta)\
**Post date:** [May 1, 2018, 8:16am UTC](https://discuss.elastic.co/t/how-to-limit-the-http-output-from-logstash/130099/1 "2018-05-01T08:16:59Z")

</div>

I have a pipeline which uses kafka as i/p and HTTP as o/p.  
I want to limit the HTTP requests that are sent from logstash o/p.  
Is there a way to do that?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [May 1, 2018, 6:14pm UTC](https://discuss.elastic.co/t/how-to-limit-the-http-output-from-logstash/130099/2 "2018-05-01T18:14:10Z")

</div>

What is the desired behavior if more events arrive to the Kafka topic than the allowed Logstash output rate? Queue up the events in Kafka for later processing or drop overflow events in Logstash?

---

<div class="post-metadata">

**Author:** ![santoshgupta](https://avatars.discourse-cdn.com/v4/letter/s/2bfe46/32.png) [@santoshgupta](https://discuss.elastic.co/u/santoshgupta)\
**Post date:** [May 2, 2018, 12:32pm UTC](https://discuss.elastic.co/t/how-to-limit-the-http-output-from-logstash/130099/3 "2018-05-02T12:32:08Z")

</div>

I don't want to drop any event. I just want to limit the output rate, so that when bursts of input comes, they don't overload my HTTP servers.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [May 2, 2018, 12:35pm UTC](https://discuss.elastic.co/t/how-to-limit-the-http-output-from-logstash/130099/4 "2018-05-02T12:35:10Z")

</div>

Perhaps the throttle filter in combination with a sleep filter will do?

> <https://github.com/elastic/logstash/issues/4726>

---

<div class="post-metadata">

**Author:** ![santoshgupta](https://avatars.discourse-cdn.com/v4/letter/s/2bfe46/32.png) [@santoshgupta](https://discuss.elastic.co/u/santoshgupta)\
**Post date:** [May 3, 2018, 11:17am UTC](https://discuss.elastic.co/t/how-to-limit-the-http-output-from-logstash/130099/5 "2018-05-03T11:17:27Z")

</div>

Would using "queue.max\_events" solve the issue?  
As per the documentation -  
"When the queue is full, Logstash puts back pressure on the inputs to stall data flowing into Logstash. This mechanism helps Logstash control the rate of data flow at the input stage without overwhelming outputs like Elasticsearch."

Ref. - [https://www.elastic.co/guide/en/logstash/current/persistent-queues.html](https://www.elastic.co/guide/en/logstash/current/persistent-queues.html)

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [May 3, 2018, 11:21am UTC](https://discuss.elastic.co/t/how-to-limit-the-http-output-from-logstash/130099/6 "2018-05-03T11:21:47Z")

</div>

`queue.max_events` doesn't provide general rate limiting.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 31, 2018, 11:21am UTC](https://discuss.elastic.co/t/how-to-limit-the-http-output-from-logstash/130099/7 "2018-05-31T11:21:58Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.