Json filter not parsing AWS WAF Logs

Elastic Stack Logstash
1

Hello Community,

I having issue with parsing AWS Waf logs using Logstash filter plugin, here is the breakdown

I'm pulling logs from AWS using kinesis input and the I'm filtering the log message using :-

filter {
  json {
    source => "message"
  }

  mutate {
    add_field => {
      "action" => "%{[logEvents][0][message][action]}"
      "aws_kinesis_stream" => "%{[logStream]}"
      "aws_log_group" => "%{[logGroup]}"
      "client_ip" => "%{[logEvents][0][message][httpRequest][clientIp]}"
      "country" => "%{[logEvents][0][message][httpRequest][country]}"
      "http_method" => "%{[logEvents][0][message][httpRequest][httpMethod]}"
    }
  }
}

the output is like this:-

      "http_method" => "%{[logEvents][0][message][httpRequest][httpMethod]}",
      "aws_kinesis_stream" => "ap-northeast-1_WebACL1_10",
      "action" => "%{[logEvents][0][message][action]}",
      "client_ip" => "%{[logEvents][0][message][httpRequest][clientIp]}",
      "logStream" => "ap-northeast-1_WebACL1_10"
      "aws_log_group" => "aws-waf-logs-test"
      "@timestamp" => 2023-11-21T13:25:31.879362082Z,
      "@version" => "1",

it parsed some fields and not parsed others; my goal is to parse the fields in a format like this: -

"action"       =>         "BLOCK"
 "client_ip"   =>         "66.66.6.6"
  "http_method" =>  "GET"

HERE IS THE MESSAGE I WANT TO PARSE

{\"messageType\":\"DATA_MESSAGE\",\"owner\":\"678447987118\",\"logGroup\":\"aws-waf-logs-test\",\"logStream\":\"ap-northeast-1_WebACL1_34\",\"subscriptionFilters\":[\"cloud-kinesis\"],\"logEvents\":[{\"id\":\"37924048000174373854086603998126409963848020333219479552\",\"timestamp\":1700573127156,\"message\":\"{\\\"timestamp\\\":1700573127156,\\\"formatVersion\\\":1,\\\"webaclId\\\":\\\"arn:aws:wafv2:ap-northeast-1:678447987118:regional/webacl/WebACL1/befbc863-6844-4275-b379-d4b88ccb5932\\\",\\\"terminatingRuleId\\\":\\\"Rule1\\\",\\\"terminatingRuleType\\\":\\\"REGULAR\\\",\\\"action\\\":\\\"BLOCK\\\",\\\"terminatingRuleMatchDetails\\\":[],\\\"httpSourceName\\\":\\\"ALB\\\",\\\"httpSourceId\\\":\\\"678447987118-app/Test-loadblancer1/472346ad8846e266\\\",\\\"ruleGroupList\\\":[{\\\"ruleGroupId\\\":\\\"AWS#AWSManagedRulesAmazonIpReputationList\\\",\\\"terminatingRule\\\":null,\\\"nonTerminatingMatchingRules\\\":[],\\\"excludedRules\\\":null,\\\"customerConfig\\\":null}],\\\"rateBasedRuleList\\\":[],\\\"nonTerminatingMatchingRules\\\":[],\\\"requestHeadersInserted\\\":null,\\\"responseCodeSent\\\":null,\\\"httpRequest\\\":{\\\"clientIp\\\":\\\"66.66.6.6\\\",\\\"country\\\":\\\"US\\\",\\\"headers\\\":[{\\\"name\\\":\\\"Host\\\",\\\"value\\\":\\\"test-loadblancer1-1599497578.ap-northeast-1.elb.amazonaws.com\\\"},{\\\"name\\\":\\\"Connection\\\",\\\"value\\\":\\\"keep-alive\\\"},{\\\"name\\\":\\\"Cache-Control\\\",\\\"value\\\":\\\"max-age=0\\\"},{\\\"name\\\":\\\"Upgrade-Insecure-Requests\\\",\\\"value\\\":\\\"1\\\"},{\\\"name\\\":\\\"User-Agent\\\",\\\"value\\\":\\\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36\\\"},{\\\"name\\\":\\\"Accept\\\",\\\"value\\\":\\\"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7\\\"},{\\\"name\\\":\\\"Accept-Encoding\\\",\\\"value\\\":\\\"gzip, deflate\\\"},{\\\"name\\\":\\\"Accept-Language\\\",\\\"value\\\":\\\"en-GB,en-US;q=0.9,en;q=0.8\\\"},{\\\"name\\\":\\\"If-None-Match\\\",\\\"value\\\":\\\"\\\\\\\"d-60a06e417d153\\\\\\\"\\\"},{\\\"name\\\":\\\"If-Modified-Since\\\",\\\"value\\\":\\\"Mon, 13 Nov 2023 11:22:53 GMT\\\"}],\\\"uri\\\":\\\"/\\\",\\\"args\\\":\\\"\\\",\\\"httpVersion\\\":\\\"HTTP/1.1\\\",\\\"httpMethod\\\":\\\"GET\\\",\\\"requestId\\\":\\\"1-655cafc7-639395cb023d5048126fd114\\\"}}\"}]}",

MY FULL LOGSTASH CONFIGURATION

input {
  kinesis {
    kinesis_stream_name => "cloud-kinesis"
    region => "ap-northeast-1"
    profile => "default"
    codec => "gzip_lines"
  }
}

filter {
  json {
    source => "message"
  }

  mutate {
    add_field => {
      "action" => "%{[logEvents][0][message][action]}"
      "aws_kinesis_stream" => "%{[logStream]}"
      "aws_log_group" => "%{[logGroup]}"
      "client_ip" => "%{[logEvents][0][message][httpRequest][clientIp]}"
      "country" => "%{[logEvents][0][message][httpRequest][country]}"
      "http_method" => "%{[logEvents][0][message][httpRequest][httpMethod]}"
    }
  }
}


output {
  stdout { codec => rubydebug }

Can one help me please

thanks.

2

That is not valid JSON, so a you cannot use a json filter to parse it. The quotes are not balanced and it has a trailing comma.

3

Hey @Badger thanks for the replay.

So, what filter I can use in order to parse this message.

Also, bro can I use the gsub filter in Logstash to remove the unnecessary characters and make the message a valid JSON.

4

Yes, you can. If you do not do that you will have to write your own parser in a ruby filter.

5

Hello @Badger thanks for the replay I finnally solved this issue by using codec => cloudwatch_logs plugin which decoded my logs.

6

@Badger hey bro can you also help me this issue https://discuss.elastic.co/t/profile-file-cannot-be-null-logstash-error/348224?u=laale1

thanks

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.