Grok expression works, but causes logstash config check to fail. Special characters in log an issue?

[hueyg]

I have a grok expression for an application where the log contains square brackets, curly brackets, and pipe characters. The expression works in the grok debugger but fails when I use it in the logstash filter.

Here is a sample of the data:

[2018-11-09 10:30:25,698]  INFO {AUDIT_LOG}-  Initiator : J00168813@carbon.super | Action : Login | Target : ApplicationAuthenticationFramework | Data : { "ContextIdentifier" : "0010e177-ebfa-43a6-9b8c-28ed64370687","AuthenticatedUser" : "J00168813@carbon.super","AuthenticatedUserTenantDomain" : "null","ServiceProviderName" : "BanAdmin_PROD","RequestType" : "cassso","RelyingParty" : "BanAdmin_PROD","AuthenticatedIdPs" : "eyJ0eXAiOiJKV1QiLCAiYWxnIjoibm9uZSJ9.eyJpc3MiOiJ3c28yIiwiZXhwIjoxNTQxNzgxMDI1NjgwMzAwMCwiaWF0IjoxNTQxNzgxMDI1NjgwLCJpZHBzIjpbeyJpZHAiOiJMT0NBTCIsImF1dGhlbnRpY2F0b3IiOiJCYXNpY0F1dGhlbnRpY2F0b3IifV19." } | Result : Success

Here is my grok expression. Notice the special characters. I have another log file with square brackets which works fine. I know I have a mix of escape characters but this is after way too many tries so forgive my inconsistent attempts:

		%{TIMESTAMP_ISO8601:date}\]\s\sINFO {AUDIT_LOG}-\s\sInitiator : %{WORD:user}@carbon.super \| Action : %{WORD:action} \| Target : %{WORD:framework} \| Data : { "ContextIdentifier" : %{QUOTEDSTRING:context},"AuthenticatedUser" : %{QUOTEDSTRING:fulluser},"AuthenticatedUserTenantDomain" : %{QUOTEDSTRING:tenant},"ServiceProviderName" : %{QUOTEDSTRING:serviceprovider},"RequestType" : %{QUOTEDSTRING:request},"RelyingParty" : %{QUOTEDSTRING:relyparty},"AuthenticatedIdPs" : %{QUOTEDSTRING:authenticatedidps} } \| Result : %{WORD:result}

FInally here is one version of the filter I am attempting:

 if [type] == "wso2audit" {
grok {
  match => "message" => ["%{TIMESTAMP_ISO8601:date}\]\s\sINFO {AUDIT_LOG}-\s\sInitiator : %{WORD:user}@carbon.super \| Action : %{WORD:action} \| Target : %{WORD:framework} \| Data : { "ContextIdentifier" : %{QUOTEDSTRING:context},"AuthenticatedUser" : %{QUOTEDSTRING:fulluser},"AuthenticatedUserTenantDomain" : %{QUOTEDSTRING:tenant},"ServiceProviderName" : %{QUOTEDSTRING:serviceprovider},"RequestType" : %{QUOTEDSTRING:request},"RelyingParty" : %{QUOTEDSTRING:relyparty},"AuthenticatedIdPs" : %{QUOTEDSTRING:authenticatedidps} } \| Result : %{WORD:result}"}]
  remove_field => "framework"
  remove_field => "tenant"
  remove_field => "relyparty"
  remove_field => "authenticatedips"
}

}

As you can see I am only interested in parsing out few of the fields and want to ignore several. Appreciate any advice as I am losing patience.

[Makra (Makara)]

Hi

@hueyg

The GROK is okay, you just need to escape the double quotes for Logstash to work. The following worked in my logstash instance after escaping the double quotes inside grok.

match => ["message" , "%{TIMESTAMP_ISO8601:date}\]\s\sINFO {AUDIT_LOG}-\s\sInitiator : %{WORD:user}@carbon.super \| Action : %{WORD:action} \| Target : %{WORD:framework} \| Data : { \"ContextIdentifier\" : %{QUOTEDSTRING:context},\"AuthenticatedUser\" : %{QUOTEDSTRING:fulluser},\"AuthenticatedUserTenantDomain\" : %{QUOTEDSTRING:tenant},\"ServiceProviderName\" : %{QUOTEDSTRING:serviceprovider},\"RequestType\" : %{QUOTEDSTRING:request},\"RelyingParty\" : %{QUOTEDSTRING:relyparty},\"AuthenticatedIdPs\" : %{QUOTEDSTRING:authenticatedidps} } \| Result : %{WORD:result}"]

[Album] Imgur: The magic of the Internet

Also use mutate filter to remove the fields.

mutate {
         remove_field => ["framework","tenant","relyparty","authenticatedips" ]
       }