# How logstash throttle filter works with auto scaling and pipeline workers? **URL:** \ **Category:** Logstash\ **Created:** [May 1, 2018, 8:43pm UTC](https://discuss.elastic.co/t/how-logstash-throttle-filter-works-with-auto-scaling-and-pipeline-workers/130161 "2018-05-01T20:43:39Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Debarghya\_Chakrabort](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/debarghya_chakrabort/32/30659_2.png) [@Debarghya\_Chakrabort](https://discuss.elastic.co/u/Debarghya_Chakrabort)\ **Post date:** [May 1, 2018, 8:43pm UTC](https://discuss.elastic.co/t/how-logstash-throttle-filter-works-with-auto-scaling-and-pipeline-workers/130161/1 "2018-05-01T20:43:39Z")
Hello, I am using ELK stack to process our logs. Now we are going to introduce throttling by using logstash throttle filter. Right now we have 2 logstash with 2 pipeline workers each and this can can be auto scaled. Can anyone please help me to understand how this will work? Thanks ---
**Author:** ![paz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paz/32/28003_2.png) [@paz](https://discuss.elastic.co/u/paz)\ **Post date:** [May 2, 2018, 1:28pm UTC](https://discuss.elastic.co/t/how-logstash-throttle-filter-works-with-auto-scaling-and-pipeline-workers/130161/2 "2018-05-02T13:28:04Z")
Throttling works per Logstash instance, regardless of pipeline workers. So if you set throttling to e.g. 100 events/sec, each Logstash instance will tag every event after it's initial share of 100 per second, for a total of 200/sec. The caveat here is, with multiple pipeline workers and high throughput you might experience slight deviations in the throttling rate. E.g. if you have 100k events/sec and want to throttle it to 50k/sec, you might observe occasions of 50,200 events passing before they are throttled or something like that. ---
**Author:** ![Debarghya\_Chakrabort](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/debarghya_chakrabort/32/30659_2.png) [@Debarghya\_Chakrabort](https://discuss.elastic.co/u/Debarghya_Chakrabort)\ **Post date:** [May 2, 2018, 4:05pm UTC](https://discuss.elastic.co/t/how-logstash-throttle-filter-works-with-auto-scaling-and-pipeline-workers/130161/3 "2018-05-02T16:05:46Z")
Thanks Paz for the clarification. "each Logstash instance will tag every event after it's initial share of 100 per second, for a total of 200/sec" - you mean each logstash pipeline workers? I am "ok" with the deviation but my concern is auto scaling. Suppose I have set throttling for 100events/sec and 2 instances of logstash which basically means 200events/sec (50events/sec/pipelineworkers as I am running this on 2 core machine) as they dont have any share memory. Now if my logstash cluster scale up to 4 nodes how can I ensure throttling of 200events/sec? ---
**Author:** ![paz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paz/32/28003_2.png) [@paz](https://discuss.elastic.co/u/paz)\ **Post date:** [May 3, 2018, 12:39pm UTC](https://discuss.elastic.co/t/how-logstash-throttle-filter-works-with-auto-scaling-and-pipeline-workers/130161/4 "2018-05-03T12:39:21Z")
> [@Debarghya\_Chakrabort](#): > > "each Logstash instance will tag every event after it's initial share of 100 per second, for a total of 200/sec" - you mean each logstash pipeline workers? No, per instance (or VM if you wanna think it that way) regardless of pipeline workers. If you set it to 100/sec, it will allow 100/sec whether you have 1 worker or 4 workers or whatever. Now, regarding the automatic adaptation it's not that straightforward. You'd need to adapt your configuration on-the-fly, probably by pushing new config files and using the _automatic reload_ feature.\ So you'd need to calculate `total expected rate / number of instances` as a throttle rate for each instance and change the config value each time a new instance is created or killed. Depending on your workflow and provisioning tools, it can range from pretty easy to slightly cumbersome. ---
**Author:** ![Debarghya\_Chakrabort](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/debarghya_chakrabort/32/30659_2.png) [@Debarghya\_Chakrabort](https://discuss.elastic.co/u/Debarghya_Chakrabort)\ **Post date:** [May 3, 2018, 1:14pm UTC](https://discuss.elastic.co/t/how-logstash-throttle-filter-works-with-auto-scaling-and-pipeline-workers/130161/5 "2018-05-03T13:14:34Z")
Thank You. This is very helpful. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 31, 2018, 1:14pm UTC](https://discuss.elastic.co/t/how-logstash-throttle-filter-works-with-auto-scaling-and-pipeline-workers/130161/6 "2018-05-31T13:14:42Z")
Closed on May 31, 2018, 1:14pm UTC This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.