How do drop logs from being forwarded? My drop rule doesn't seem to be working

[feo13]

I'm ingesting AWS WAF logs and would like to drop the 'ALLOW' logs. I have the following filter in place but it doesn't seem to be working:

filter { 

  if "\"action\":\"ALLOW\"" in [message] {
    drop {}
  }

  if [type] == "waf-log" { 
   json { 
        source => "message" 
  } 
  date { 
        match => [ "[timestamp]", "UNIX_MS" ] 
  } 
  geoip { 
        source => [ "[httpRequest][clientIp]" ] 
        target => geoip 
  } 
  } 


} 

The logs are ugly but near the top of this mess is what I'm trying to match on ("action":"ALLOW") :

{"timestamp":1691284267528,"formatVersion":1,"webaclId":"arn:aws:wafv2:us-east-1:13:regional/webacl/testhaproxy-pub/348b1a33-94f8-4090-98c3-2d58fb0802f2","terminatingRuleId":"Default_Action","terminatingRuleType":"REGULAR","action":"ALLOW","terminatingRuleMatchDetails":[],"httpSourceName":"ALB","httpSourceId":"13-app/test/7646e40a4e7f47c8","ruleGroupList":[{"ruleGroupId":"arn:aws:wafv2:us-east-1:13:regional/rulegroup/Community_sourced_IP_block_list/cfa0b98a-c8f9-4832-9e46-33e3363d4902","terminatingRule":null,"nonTerminatingMatchingRules":[],"excludedRules":null,"customerConfig":null},{"ruleGroupId":"AWS#AWSManagedRulesKnownBadInputsRuleSet","terminatingRule":null,"nonTerminatingMatchingRules":[],"excludedRules":null,"customerConfig":null}],"rateBasedRuleList":[],"nonTerminatingMatchingRules":[],"requestHeadersInserted":null,"responseCodeSent":null,"httpRequest":{"clientIp":"8.8.8.8","country":"US","headers":[{"name":"Host","value":"test.example.com"},{"name":"User-Agent","value":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36"},{"name":"X-Amz-Cf-Id","value":"UhoYRGs6vYv9voCY1PMYwRnyT9K-PddELJnhH2AwS9Hzm1AHwTcJJg=="},{"name":"Connection","value":"Keep-Alive"},{"name":"Content-Length","value":"1420"},{"name":"Via","value":"2.0 cf067f00f2e5b21f25d2ffca9aa60006.cloudfront.net (CloudFront)"},{"name":"X-Forwarded-For","value":"4.4.4.4"},{"name":"Accept-Language","value":"en-US,en;q=0.9"},{"name":"Accept","value":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"},{"name":"Referer","value":"https://test.example.com/s3/2/3"},{"name":"Accept-Encoding","value":"gzip, deflate, br"},{"name":"cache-control","value":"max-age=0"},{"name":"sec-ch-ua","value":"\"Not/A)Brand\";v=\"99\", \"Google Chrome\";v=\"115\", \"Chromium\";v=\"115\""},{"name":"sec-ch-ua-mobile","value":"?0"},{"name":"sec-ch-ua-platform","value":"\"Windows\""},{"name":"upgrade-insecure-requests","value":"1"},{"name":"origin","value":"https://test.example.com"},{"name":"content-type","value":"multipart/form-data; boundary=----WebKitFormBoundaryfcoYCAy2fmiB4igH"},{"name":"sec-fetch-site","value":"same-origin"},{"name":"sec-fetch-mode","value":"navigate"},{"name":"sec-fetch-user","value":"?1"},{"name":"sec-fetch-dest","value":"document"},{"name":"CloudFront-Is-Mobile-Viewer","value":"false"},{"name":"CloudFront-Is-Tablet-Viewer","value":"false"},{"name":"CloudFront-Is-SmartTV-Viewer","value":"false"},{"name":"CloudFront-Is-Desktop-Viewer","value":"true"},{"name":"CloudFront-Viewer-Country","value":"US"},{"name":"CloudFront-Forwarded-Proto","value":"https"},{"name":"CloudFront-Viewer-ASN","value":"22773"}],"uri":"/s3/2/3","args":"","httpVersion":"HTTP/1.1","httpMethod":"POST","requestId":"1-64cef32b-644627a2000354c90c5e7329"},"requestBodySize":1420,"requestBodySizeInspectedByWAF":1420}

How does that need to get set up so it drops these logs properly?

A side question -- is there a way I can delete the existing "action":"ALLOW" logs from all indicies?

Thank you!

[Ljapunov]

Hi,
actually I do not think that what you search is in the [message] field. Try this one:

if  [action] == "ALLOW" {
    drop {}
  }

And to your side question: Yes you can use _delete_by_query API.

[Badger]

Doing the drop after parsing the JSON results in a lot of wasted work. It would be more efficient to match the [message] field than the [action] field. And if these are firewall logs I would expect the majority of events to have ALLOW, so it is a big optimisation.

[feo13]

Thank you both for the replies. I tried changing it to 'if [action] == "ALLOW"', but I'm still seeing allow logs coming in. Here's the updated filter:

filter {

  if [action] == "ALLOW" {
    drop {}
  }

  if [type] == "waf-log" {
   json {
        source => "message"
  }
  date {
        match => [ "[timestamp]", "UNIX_MS" ]
  }
  geoip {
        source => [ "[httpRequest][clientIp]" ]
        target => geoip
  }
  }

}

But I expand one of the logs, I do see that field so I'm not sure what's going on.

And @Badger, you're right -- ~97% of the logs are allows so getting this right will make a huge impact.

Thanks again!

[Badger]

@feo13

if "\"action\":\"ALLOW\""

Are you sure all the whitespace is stripped from this type of log? Looking at AWF example logs some do, some do not. You could try '"action":\s*"ALLOW"'.