How logstash throttle filter works with auto scaling and pipeline workers?

Elastic Stack Logstash
1

Hello,

I am using ELK stack to process our logs. Now we are going to introduce throttling by using logstash throttle filter. Right now we have 2 logstash with 2 pipeline workers each and this can can be auto scaled.
Can anyone please help me to understand how this will work?

Thanks

2

Throttling works per Logstash instance, regardless of pipeline workers.
So if you set throttling to e.g. 100 events/sec, each Logstash instance will tag every event after it's initial share of 100 per second, for a total of 200/sec.

The caveat here is, with multiple pipeline workers and high throughput you might experience slight deviations in the throttling rate. E.g. if you have 100k events/sec and want to throttle it to 50k/sec, you might observe occasions of 50,200 events passing before they are throttled or something like that.

1 Like
3

Thanks Paz for the clarification.

"each Logstash instance will tag every event after it's initial share of 100 per second, for a total of 200/sec" - you mean each logstash pipeline workers?

I am "ok" with the deviation but my concern is auto scaling. Suppose I have set throttling for 100events/sec and 2 instances of logstash which basically means 200events/sec (50events/sec/pipelineworkers as I am running this on 2 core machine) as they dont have any share memory. Now if my logstash cluster scale up to 4 nodes how can I ensure throttling of 200events/sec?

4

No, per instance (or VM if you wanna think it that way) regardless of pipeline workers. If you set it to 100/sec, it will allow 100/sec whether you have 1 worker or 4 workers or whatever.

Now, regarding the automatic adaptation it's not that straightforward. You'd need to adapt your configuration on-the-fly, probably by pushing new config files and using the automatic reload feature.\

So you'd need to calculate total expected rate / number of instances as a throttle rate for each instance and change the config value each time a new instance is created or killed. Depending on your workflow and provisioning tools, it can range from pretty easy to slightly cumbersome.

1 Like
5

Thank You. This is very helpful.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.