Logstash output send rate

Elastic Stack Logstash
1

Hi Elastic Team,

Does Logstash have a feature that allows us to set how many events are sent to remotes, for example outputted to Elasticsearch, for every particular time interval?

What we are trying to do is something like this: have no more than X number of events or Y MB of data sent by Logstash every N time interval - a bit like throttling.

I still want all the events to be sent just at a controlled rate.

I've tried using the throttle plugin but am not sure whether it fits the purpose.

Cheers,

2

Throttle is not what you want.
You should run 5.2 and then use the Monitoring functionality.

3

Hi Mark,

Thanks for your reply.

Maybe I've missed something but x-pack monitoring for logstash only gives me the rate values but doesn't really give me the ability to tweak the rates themselves.

Say I notice that Logstash is sending too many events per second, how can I tell Logstash to "calm down" a bit and do not exceed x amount of events per second or x MB per second.

4

I misread that sorry.

If you are sending to ES then it'll tell LS to slow down, which it will. I don't know how to do that for other outpus.

5

Yep, I'm sending to ES.

That is interesting, I did not know that ES tells LS to slow down.

Do you have links to any of the documentations for that?

Or do you know concrete numbers like by how much ES tells LS to slow down and how it's doing that and whether it is configurable?

6

ES dictates it all, see https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#_retry_policy for details.

7

Thanks Mark, I've checked the link but the retry policy only describes the behaviour when there is a disconnect.

I've looked further into the adaptive throttling feature of ES which determines the indexing/merge IO rate of ES when the load is high and when the load is low. So hypothetically, if when the load is high, the merge IO rate has been dynamically determined to be 200MB/s, does that mean roughly if I have 10 Logstash instances each of them will only be sending 20 MB/s?

And if I really wanted to, I could set indices.store.throttle.max_bytes_per_sec (I understand this is not recommended) explicitly that would indirectly limit the send rate of logstash during high load?

Also a side question: before sending data requests/docs to ES, does Logstash send some sort of handshake or status requests or permission requests prior to sending data to know whether or not to send the docs over to ES?

8

If ES is too busy it will provide a 429, then the retry policy starts. That's as complicated as it gets (good or bad).

No. That only applies for older versions of ES, newer versions auto throttle.

It's a typical http connection request.

9

Many thanks Mark

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.