Throttle filter is not throttling correctly

[holobolo0815]

With reference to Throttle filter: Notify on throttle

When doing this:

if [srcip] and [dstip] and [dstport] {
  throttle { # decorates event if OUTSIDE bounds
    key => "%{%filter_matched}-%{srcip}-%{dstip}-%{dstport}"
    before_count => -1
    after_count => 749 # same as below - 1
    period => 60
    max_age => 300
    add_tag => "throttle"
  }
  throttle { # run again to tag what should actually be dropped
    key => "%{%filter_matched}-%{srcip}-%{dstip}-%{dstport}"
    before_count => -1
    after_count => 750 # same as above + 1
    period => 60
    max_age => 300
    add_tag => "throttleDrop"
  }
  if "throttleDrop" in [tags] {
    drop {}
  }
}

(Yes, the name of the variable "%filter_matched" begins with a percent sign)

The following still happens:

Expected: At most 1 event per minute with only "throttle" tag set when event count using the specified key exceeds 750.

Actual: Events that should be dropped are in fact not.

How do you debug the workings of the throttle filter? (like showing what the currently used keys are etc.)

UPDATE By looking at the source of throttle filter you see that there is interesting debug output

You can temporarily enable it using (https://www.elastic.co/guide/en/logstash/current/logging.html)

curl -XPUT 'localhost:9600/_node/logging?pretty' -H 'Content-Type: application/json' -d'
{
	"logger.logstash.filters.throttle" : "DEBUG"
}
'

(Default log level is INFO) Beware of disk space consumption! Maybe you just want to enable it for a few minutes and then disable again.

curl -XPUT 'localhost:9600/_node/logging?pretty' -H 'Content-Type: application/json' -d'
{
	"logger.logstash.filters.throttle" : "INFO"
}
'

After that check the logstash logs at /var/log/logstash.

It will show stuff like

filters/LogStash::Filters::Throttle: counter incremented {:key=>"abcd-172.16.1.2-172.31.1.2-53", :epoch=>1611770594, :timeslot=>1611770591, :count=>1}

I'll try to get to the bottom of it like that. Feel free for comments though!