# Json filter not parsing AWS WAF Logs **URL:** \ **Category:** Logstash\ **Created:** [November 21, 2023, 1:39pm UTC](https://discuss.elastic.co/t/json-filter-not-parsing-aws-waf-logs/347639 "2023-11-21T13:39:15Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![laale1](https://avatars.discourse-cdn.com/v4/letter/l/90ced4/32.png) [@laale1](https://discuss.elastic.co/u/laale1)\ **Post date:** [November 21, 2023, 1:39pm UTC](https://discuss.elastic.co/t/json-filter-not-parsing-aws-waf-logs/347639/1 "2023-11-21T13:39:15Z")
Hello Community, I having issue with parsing AWS Waf logs using Logstash filter plugin, here is the breakdown I'm pulling logs from AWS using kinesis input and the I'm filtering the log message using :- ```auto filter { json { source => "message" } mutate { add_field => { "action" => "%{[logEvents][0][message][action]}" "aws_kinesis_stream" => "%{[logStream]}" "aws_log_group" => "%{[logGroup]}" "client_ip" => "%{[logEvents][0][message][httpRequest][clientIp]}" "country" => "%{[logEvents][0][message][httpRequest][country]}" "http_method" => "%{[logEvents][0][message][httpRequest][httpMethod]}" } } } ``` the output is like this:- ```auto "http_method" => "%{[logEvents][0][message][httpRequest][httpMethod]}", "aws_kinesis_stream" => "ap-northeast-1_WebACL1_10", "action" => "%{[logEvents][0][message][action]}", "client_ip" => "%{[logEvents][0][message][httpRequest][clientIp]}", "logStream" => "ap-northeast-1_WebACL1_10" "aws_log_group" => "aws-waf-logs-test" "@timestamp" => 2023-11-21T13:25:31.879362082Z, "@version" => "1", ``` it parsed some fields and not parsed others; my goal is to parse the fields in a format like this: - ```auto "action" => "BLOCK" "client_ip" => "66.66.6.6" "http_method" => "GET" ``` HERE IS THE MESSAGE I WANT TO PARSE ```auto {\"messageType\":\"DATA_MESSAGE\",\"owner\":\"678447987118\",\"logGroup\":\"aws-waf-logs-test\",\"logStream\":\"ap-northeast-1_WebACL1_34\",\"subscriptionFilters\":[\"cloud-kinesis\"],\"logEvents\":[{\"id\":\"37924048000174373854086603998126409963848020333219479552\",\"timestamp\":1700573127156,\"message\":\"{\\\"timestamp\\\":1700573127156,\\\"formatVersion\\\":1,\\\"webaclId\\\":\\\"arn:aws:wafv2:ap-northeast-1:678447987118:regional/webacl/WebACL1/befbc863-6844-4275-b379-d4b88ccb5932\\\",\\\"terminatingRuleId\\\":\\\"Rule1\\\",\\\"terminatingRuleType\\\":\\\"REGULAR\\\",\\\"action\\\":\\\"BLOCK\\\",\\\"terminatingRuleMatchDetails\\\":[],\\\"httpSourceName\\\":\\\"ALB\\\",\\\"httpSourceId\\\":\\\"678447987118-app/Test-loadblancer1/472346ad8846e266\\\",\\\"ruleGroupList\\\":[{\\\"ruleGroupId\\\":\\\"AWS#AWSManagedRulesAmazonIpReputationList\\\",\\\"terminatingRule\\\":null,\\\"nonTerminatingMatchingRules\\\":[],\\\"excludedRules\\\":null,\\\"customerConfig\\\":null}],\\\"rateBasedRuleList\\\":[],\\\"nonTerminatingMatchingRules\\\":[],\\\"requestHeadersInserted\\\":null,\\\"responseCodeSent\\\":null,\\\"httpRequest\\\":{\\\"clientIp\\\":\\\"66.66.6.6\\\",\\\"country\\\":\\\"US\\\",\\\"headers\\\":[{\\\"name\\\":\\\"Host\\\",\\\"value\\\":\\\"test-loadblancer1-1599497578.ap-northeast-1.elb.amazonaws.com\\\"},{\\\"name\\\":\\\"Connection\\\",\\\"value\\\":\\\"keep-alive\\\"},{\\\"name\\\":\\\"Cache-Control\\\",\\\"value\\\":\\\"max-age=0\\\"},{\\\"name\\\":\\\"Upgrade-Insecure-Requests\\\",\\\"value\\\":\\\"1\\\"},{\\\"name\\\":\\\"User-Agent\\\",\\\"value\\\":\\\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36\\\"},{\\\"name\\\":\\\"Accept\\\",\\\"value\\\":\\\"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7\\\"},{\\\"name\\\":\\\"Accept-Encoding\\\",\\\"value\\\":\\\"gzip, deflate\\\"},{\\\"name\\\":\\\"Accept-Language\\\",\\\"value\\\":\\\"en-GB,en-US;q=0.9,en;q=0.8\\\"},{\\\"name\\\":\\\"If-None-Match\\\",\\\"value\\\":\\\"\\\\\\\"d-60a06e417d153\\\\\\\"\\\"},{\\\"name\\\":\\\"If-Modified-Since\\\",\\\"value\\\":\\\"Mon, 13 Nov 2023 11:22:53 GMT\\\"}],\\\"uri\\\":\\\"/\\\",\\\"args\\\":\\\"\\\",\\\"httpVersion\\\":\\\"HTTP/1.1\\\",\\\"httpMethod\\\":\\\"GET\\\",\\\"requestId\\\":\\\"1-655cafc7-639395cb023d5048126fd114\\\"}}\"}]}", ``` MY FULL LOGSTASH CONFIGURATION ```auto input { kinesis { kinesis_stream_name => "cloud-kinesis" region => "ap-northeast-1" profile => "default" codec => "gzip_lines" } } filter { json { source => "message" } mutate { add_field => { "action" => "%{[logEvents][0][message][action]}" "aws_kinesis_stream" => "%{[logStream]}" "aws_log_group" => "%{[logGroup]}" "client_ip" => "%{[logEvents][0][message][httpRequest][clientIp]}" "country" => "%{[logEvents][0][message][httpRequest][country]}" "http_method" => "%{[logEvents][0][message][httpRequest][httpMethod]}" } } } output { stdout { codec => rubydebug } ``` Can one help me please thanks. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [November 21, 2023, 5:20pm UTC](https://discuss.elastic.co/t/json-filter-not-parsing-aws-waf-logs/347639/2 "2023-11-21T17:20:33Z")
> [@laale1](#): > > HERE IS THE MESSAGE I WANT TO PARSE > > ```auto > {\"messageType\":\"DATA_MESSAGE\",\"owner\":\"678447987118\",\"logGroup\":\"aws-waf-logs-test\",\"logStream\":\"ap-northeast-1_WebACL1_34\",\"subscriptionFilters\":[\"cloud-kinesis\"],\"logEvents\":[{\"id\":\"37924048000174373854086603998126409963848020333219479552\",\"timestamp\":1700573127156,\"message\":\"{\\\"timestamp\\\":1700573127156,\\\"formatVersion\\\":1,\\\"webaclId\\\":\\\"arn:aws:wafv2:ap-northeast-1:678447987118:regional/webacl/WebACL1/befbc863-6844-4275-b379-d4b88ccb5932\\\",\\\"terminatingRuleId\\\":\\\"Rule1\\\",\\\"terminatingRuleType\\\":\\\"REGULAR\\\",\\\"action\\\":\\\"BLOCK\\\",\\\"terminatingRuleMatchDetails\\\":[],\\\"httpSourceName\\\":\\\"ALB\\\",\\\"httpSourceId\\\":\\\"678447987118-app/Test-loadblancer1/472346ad8846e266\\\",\\\"ruleGroupList\\\":[{\\\"ruleGroupId\\\":\\\"AWS#AWSManagedRulesAmazonIpReputationList\\\",\\\"terminatingRule\\\":null,\\\"nonTerminatingMatchingRules\\\":[],\\\"excludedRules\\\":null,\\\"customerConfig\\\":null}],\\\"rateBasedRuleList\\\":[],\\\"nonTerminatingMatchingRules\\\":[],\\\"requestHeadersInserted\\\":null,\\\"responseCodeSent\\\":null,\\\"httpRequest\\\":{\\\"clientIp\\\":\\\"66.66.6.6\\\",\\\"country\\\":\\\"US\\\",\\\"headers\\\":[{\\\"name\\\":\\\"Host\\\",\\\"value\\\":\\\"test-loadblancer1-1599497578.ap-northeast-1.elb.amazonaws.com\\\"},{\\\"name\\\":\\\"Connection\\\",\\\"value\\\":\\\"keep-alive\\\"},{\\\"name\\\":\\\"Cache-Control\\\",\\\"value\\\":\\\"max-age=0\\\"},{\\\"name\\\":\\\"Upgrade-Insecure-Requests\\\",\\\"value\\\":\\\"1\\\"},{\\\"name\\\":\\\"User-Agent\\\",\\\"value\\\":\\\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36\\\"},{\\\"name\\\":\\\"Accept\\\",\\\"value\\\":\\\"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7\\\"},{\\\"name\\\":\\\"Accept-Encoding\\\",\\\"value\\\":\\\"gzip, deflate\\\"},{\\\"name\\\":\\\"Accept-Language\\\",\\\"value\\\":\\\"en-GB,en-US;q=0.9,en;q=0.8\\\"},{\\\"name\\\":\\\"If-None-Match\\\",\\\"value\\\":\\\"\\\\\\\"d-60a06e417d153\\\\\\\"\\\"},{\\\"name\\\":\\\"If-Modified-Since\\\",\\\"value\\\":\\\"Mon, 13 Nov 2023 11:22:53 GMT\\\"}],\\\"uri\\\":\\\"/\\\",\\\"args\\\":\\\"\\\",\\\"httpVersion\\\":\\\"HTTP/1.1\\\",\\\"httpMethod\\\":\\\"GET\\\",\\\"requestId\\\":\\\"1-655cafc7-639395cb023d5048126fd114\\\"}}\"}]}", > > ``` That is not valid JSON, so a you cannot use a json filter to parse it. The quotes are not balanced and it has a trailing comma. ---
**Author:** ![laale1](https://avatars.discourse-cdn.com/v4/letter/l/90ced4/32.png) [@laale1](https://discuss.elastic.co/u/laale1)\ **Post date:** [November 22, 2023, 6:14am UTC](https://discuss.elastic.co/t/json-filter-not-parsing-aws-waf-logs/347639/3 "2023-11-22T06:14:39Z")
Hey @Badger thanks for the replay. So, what filter I can use in order to parse this message. Also, bro can I use the `gsub` filter in Logstash to remove the unnecessary characters and make the message a valid JSON. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [November 22, 2023, 6:14pm UTC](https://discuss.elastic.co/t/json-filter-not-parsing-aws-waf-logs/347639/4 "2023-11-22T18:14:13Z")
> [@laale1](#): > > Also, bro can I use the `gsub` filter in Logstash to remove the unnecessary characters and make the message a valid JSON. Yes, you can. If you do not do that you will have to write your own parser in a ruby filter. ---
**Author:** ![laale1](https://avatars.discourse-cdn.com/v4/letter/l/90ced4/32.png) [@laale1](https://discuss.elastic.co/u/laale1)\ **Post date:** [November 29, 2023, 11:44am UTC](https://discuss.elastic.co/t/json-filter-not-parsing-aws-waf-logs/347639/5 "2023-11-29T11:44:02Z")
Hello @Badger thanks for the replay I finnally solved this issue by using codec =\> cloudwatch\_logs plugin which decoded my logs. ---
**Author:** ![laale1](https://avatars.discourse-cdn.com/v4/letter/l/90ced4/32.png) [@laale1](https://discuss.elastic.co/u/laale1)\ **Post date:** [November 29, 2023, 11:45am UTC](https://discuss.elastic.co/t/json-filter-not-parsing-aws-waf-logs/347639/6 "2023-11-29T11:45:34Z")
@Badger hey bro can you also help me this issue [https://discuss.elastic.co/t/profile-file-cannot-be-null-logstash-error/348224?u=laale1](https://discuss.elastic.co/t/profile-file-cannot-be-null-logstash-error/348224) thanks ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 27, 2023, 11:46am UTC](https://discuss.elastic.co/t/json-filter-not-parsing-aws-waf-logs/347639/7 "2023-12-27T11:46:32Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.