Throttle filter: Notify on throttle

Elastic Stack Logstash
1

When using the throttle filter, how would I go about notifying one event once the throttle has triggered?

Meaning if I allow 5 events per minute, I'd like to know somehow if there has been a 6th event and send a mail or output some kind of "special" event to ES. I don't care about subsequent events during that time period.

2

A throttle filter checks how many events have occurred in a period and if the number is outside the configured range then it decorates the event (i.e. it processes the add_tag / add_field / remove_tag / remove_field options). If you only want one event each period to not be decorated then configure before_count and after_count to have the same value.

3

Okay, let me rephrase: I want to

  1. Drop all events that do not fit into the throttle parameters (e.g. if there are more than 5 events of a certain type during an interval of 1 minute, I want the rest to be dropped)
  2. Additionally, I would like to know that this was triggered, i.e. I would like to know if stuff gets dropped that way.
4

OK, so use two throttle filters. One to tag all the events that you want to drop, and another (with before_count and after_count set to the same value) that you can conditionally route to a destination that tells you events have been throttled.

5

I'm not sure what you mean. Once I've drop{}ed the event I can't use it any longer. Do you have a tangible example of how to do this?

6

I have not tested it but you could try

filter {
    throttle {
        period => 60
        after_count => 5
        add_tag => [ "dropMe" ]
    }
    throttle {
        period => 60
        before_count => 5
        after_count => 5
        add_tag => [ "importantWhenNotPresent" ]
    }
    if "importantWhenNotPresent" not in [tags] {
        mutate {
            remove_tag => [ "importantWhenNotPresent", "dropMe" ]
            add_tag => [ "sendAlert" ]
        }
    }
    if "dropMe" in [tags] { drop {} }
}
output {
    if "sendAlert" in [tags] {
        ...
    }
}
7

Double negation... oh thanks, I hate it. :wink:

Your idea boils down to "negatively" tagging the "interesting" event in addition to it needing to be dropped.

1: importantWhenNotPresent
2: importantWhenNotPresent
3: importantWhenNotPresent
4: importantWhenNotPresent
5: importantWhenNotPresent
6: throttle
7: throttle importantWhenNotPresent
8: throttle importantWhenNotPresent
...

Maybe the following could work. It still looks like pain.

filter {
  throttle { # decorates event if OUTSIDE bounds
    key => "%{srcip}-%{dstip}-%{port}"
    before_count => -1
    after_count => 12
    period => 60
    max_age => 300
    add_tag => "throttle"
  }
  throttle { # decorates event if OUTSIDE bounds
    key => "%{srcip}-%{dstip}-%{port}"
    before_count => 12 # same as after_count of previous throttle filter
    after_count => 12 # same as before
    period => 60
    max_age => 300
    add_tag => "importantWhenNotPresent"
  }
  if "throttle" in [tags] and "importantWhenNotPresent" in [tags] {
    drop {}
  }
  # what remains are the events before throttle kicks in
  # and the first event that is throttled (which does NOT contain the "important" tag)
}

Now we should be able to filter using tag "throttle" to get the first of each set of events that was throttled. Of course you could also send an email in output{} section and whatnot.

(Not tested yet. Did I mention it looks painful?)

8

I do not think there is going to be a pretty solution to this problem. You want to drop events but also have an event that tells you those events were dropped.

Another approach would be to have a metrics filter that counts to number of events with a "throttle" tag.

Another approach would be to use pipeline-to-pipeline with a forked path pattern. Send the events to a pipeline that applies the simple throttle to drop events, and also to a pipeline that generates the alerts. Each pipeline might be easier to follow but I suspect this is over complicating things.

1 Like
9

Just some remarks for the rest of the world.
Actually you do not need to tag every event up to the limit.
Also you need to use an offset of one in after_count between filters, otherwise you don't get the message that "sticks out".

It then becomes this:

filter {
  # prevent DoS
  # "negatively" tag the "interesting" event in addition to it needing to be dropped
  # example: before_count=>-1, after_count=>5 and 6, respectively
  #1: NOOP
  #2: NOOP
  #3: NOOP
  #4: NOOP
  #5: NOOP
  #6: throttle
  #7: throttle throttleDrop
  #8: throttle throttleDrop
 
  throttle { # decorates event if OUTSIDE bounds
    key => "%{srcip}-%{dstip}-%{port}"
    before_count => -1
    after_count => 499 # same as below - 1
    period => 60
    max_age => 300
    add_tag => "throttle"
  }
  throttle { # decorates event if OUTSIDE bounds
    key => "%{srcip}-%{dstip}-%{port}"
    before_count => -1
    after_count => 500 # same as above + 1
    period => 60
    max_age => 300
    add_tag => "throttleDrop"
  }
  if "throttleDrop" in [tags] {
    drop {}
  }
}

output {
  if "throttle" in [tags] {
    email {
      from => 'holobolo@example.com'
      to => 'holobolo@example.com'
      subject => 'Throttle Alert - %{srcip}-%{dstip}-%{port}'
    }
  }
}
1 Like
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.