Throttle key?

[oscario]

Dear all,

I want the throttle plugin to group every line (not only lines matching a specific key value).
Since a key is required and the only thing all lines have in common is at least one space I use a space for the key

Si it works to add:
key => " "

But if I use a key which will never exist in the input, i.e.:
key => "THIS_PATTERN_NEVER_MATCHES"

The throttle result seems the same...

Does any know if there is a difference between using a key which always mathes and a key which never matches? And what's best to use?
In the ruby source I cannot figure out the answer (sorry...)

More info:
I use:
throttle {
after_count => 300
period => 1
max_age => 2
key => " "
add_tag => "throttled"
}
if "throttled" in [tags] {
sleep {
time => "0.1"
}
}

We peak at around 2000 events per second and I want to throttle this to around 600/sec. The above config does this. But, as said, also with a key which never matches.

Kind regards,
Oscar.

[Badger]

I think you have slightly misunderstood the usage of key.

The normal way to use key is to have a %{} reference to some field on the event. If you had an 'method' field and you wanted to throttle events for each method separately you would use

key => "%{method}"

When the filter sees an event it does the substitution of %{method} with the value of the method field, GET, for example. It then puts this event into the GET bucket and applies the throttle. If another event has method POST, then the filter puts it into the POST bucket and decides whether the throttle parameters are applied for that bucket.

If you set the key to any constant value, such as " ", then every single event goes into the " " bucket.

[oscario]

Hi Badger,

Thank you for your response.
But that is precise what I want: every single event needs to go into the bucket so throttling is always done on all events. We don't know the events we get but do know we dont't want to go over about 600 events per second. It works but my question is: using a key which does never match seems to do the same (as opposed to using a space which always matches) and I'm wondering what best to use.

FYI: we do not use Elasticsearch as a backend (which thottles automagically).

Kind regards,
Oscar.

[Badger]

@oscario

using a key which does never match seems to do the same (as opposed to using a space which always matches) and I'm wondering what best to use.

There is no such thing as "matching". If you use a constant key, whatever it is, then every event goes into the same bucket.