How logstash throttle filter works with auto scaling and pipeline workers?


I am using ELK stack to process our logs. Now we are going to introduce throttling by using logstash throttle filter. Right now we have 2 logstash with 2 pipeline workers each and this can can be auto scaled.
Can anyone please help me to understand how this will work?


Throttling works per Logstash instance, regardless of pipeline workers.
So if you set throttling to e.g. 100 events/sec, each Logstash instance will tag every event after it's initial share of 100 per second, for a total of 200/sec.

The caveat here is, with multiple pipeline workers and high throughput you might experience slight deviations in the throttling rate. E.g. if you have 100k events/sec and want to throttle it to 50k/sec, you might observe occasions of 50,200 events passing before they are throttled or something like that.

1 Like

Thanks Paz for the clarification.

"each Logstash instance will tag every event after it's initial share of 100 per second, for a total of 200/sec" - you mean each logstash pipeline workers?

I am "ok" with the deviation but my concern is auto scaling. Suppose I have set throttling for 100events/sec and 2 instances of logstash which basically means 200events/sec (50events/sec/pipelineworkers as I am running this on 2 core machine) as they dont have any share memory. Now if my logstash cluster scale up to 4 nodes how can I ensure throttling of 200events/sec?

No, per instance (or VM if you wanna think it that way) regardless of pipeline workers. If you set it to 100/sec, it will allow 100/sec whether you have 1 worker or 4 workers or whatever.

Now, regarding the automatic adaptation it's not that straightforward. You'd need to adapt your configuration on-the-fly, probably by pushing new config files and using the automatic reload feature.\

So you'd need to calculate total expected rate / number of instances as a throttle rate for each instance and change the config value each time a new instance is created or killed. Depending on your workflow and provisioning tools, it can range from pretty easy to slightly cumbersome.

1 Like

Thank You. This is very helpful.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.