How To parse Multiline Message With Grok

[ankitdevnalkar]

I am working with AWS Cloudwatchs' GuardDuty finding events, I am getting GuardDuty event in the following format :

"message": "\n\"AWS MY_ACCOUNTID has a severity 2 GuardDuty finding type Recon:EC2/PortProbeUnprotectedPort in the ap-south-1 region.\"\n\"Finding Description:\"\n\"EC2 instance has an unprotected port which is being probed by a known malicious host.. \"\n\"For more details open the GuardDuty console at HTTPS://GUARDDUTY_FINDING_URL

I have written Grok parser as follows :

(?m).*AWS %{NUMBER:accountId} .*severity %{NUMBER:severity} .*GuardDuty finding type %{NOTSPACE:findingType} .*in the %{NOTSPACE:region}

Problem statement : I would like to parse the entire second line(EC2 instance has an unprotected port which is being probed by a known malicious host..) in a single field. I tried with .*EC2 %{GREEDYDATA: Interpretation} but it parses the rest of the whole message. It should parse till \"\n"\.

[Cad]

Hi,

I assume the log you give to us is from the stdout rubydebug so i will consider that they look like this in input :

"AWS MY_ACCOUNTID has a severity 2 GuardDuty finding type Recon:EC2/PortProbeUnprotectedPort in the ap-south-1 region."
"Finding Description:"
"EC2 instance has an unprotected port which is being probed by a known malicious host.. "
"For more details open the GuardDuty console at HTTPS://GUARDDUTY_FINDING_URL

In grok you have a pattern name QUOTEDSTRING to take all the values between quotes.
So you can use it to take an entire line

Cad.

[ankitdevnalkar]

@Cad

I assume the log you give to us is from the stdout rubydebug so i will consider that they look like this in input :

Thanks for reply @Cad , Yes that is correct, I gave it from the stdout rubydebug.

I tried with (?m).*AWS %{NUMBER:accountId} .*severity %{NUMBER:severity} .*GuardDuty finding type %{NOTSPACE:findingType} .*in the %{NOTSPACE:region} %{QUOTEDSTRING:Interpretation}

But it is not working !

[Cad]

You can't put the pattern for Interpretation after the pattern region with a space between. Replace the space with .*

(?m).*AWS %{NUMBER:accountId} .*severity %{NUMBER:severity} .*GuardDuty finding type %{NOTSPACE:findingType} .*in the %{NOTSPACE:region}.*%{QUOTEDSTRING:finding}[\n]%{QUOTEDSTRING:Interpretation}[\n]%{QUOTEDSTRING:details}

Cad.

[ankitdevnalkar]

hey @Cad that worked !!
Just one more question , now parsed data coming like :
"EC2 instance has an unprotected port which is being probed by a known malicious host.. " Note : along with quotas

But I saw it is showing in stdout rubydebug as following:
"\"EC2 instance has an unprotected port which is being probed by a known malicious host.. \""

Now, to remove the quotas do I required to replace " and \ or just replacing " will be ok. You can assume I would replace quotas after data gets parsed.

[Cad]

You just have to replace the ". The \ is here just to show that the quote is a part of the string.
You can use the gsub option of the mutate filter to replace character.

[ankitdevnalkar]

Thanks @Cad ! it was very helpful !