I have a grok expression for an application where the log contains square brackets, curly brackets, and pipe characters. The expression works in the grok debugger but fails when I use it in the logstash filter.
Here is a sample of the data:
[2018-11-09 10:30:25,698] INFO {AUDIT_LOG}- Initiator : J00168813@carbon.super | Action : Login | Target : ApplicationAuthenticationFramework | Data : { "ContextIdentifier" : "0010e177-ebfa-43a6-9b8c-28ed64370687","AuthenticatedUser" : "J00168813@carbon.super","AuthenticatedUserTenantDomain" : "null","ServiceProviderName" : "BanAdmin_PROD","RequestType" : "cassso","RelyingParty" : "BanAdmin_PROD","AuthenticatedIdPs" : "eyJ0eXAiOiJKV1QiLCAiYWxnIjoibm9uZSJ9.eyJpc3MiOiJ3c28yIiwiZXhwIjoxNTQxNzgxMDI1NjgwMzAwMCwiaWF0IjoxNTQxNzgxMDI1NjgwLCJpZHBzIjpbeyJpZHAiOiJMT0NBTCIsImF1dGhlbnRpY2F0b3IiOiJCYXNpY0F1dGhlbnRpY2F0b3IifV19." } | Result : Success
Here is my grok expression. Notice the special characters. I have another log file with square brackets which works fine. I know I have a mix of escape characters but this is after way too many tries so forgive my inconsistent attempts:
%{TIMESTAMP_ISO8601:date}\]\s\sINFO {AUDIT_LOG}-\s\sInitiator : %{WORD:user}@carbon.super \| Action : %{WORD:action} \| Target : %{WORD:framework} \| Data : { "ContextIdentifier" : %{QUOTEDSTRING:context},"AuthenticatedUser" : %{QUOTEDSTRING:fulluser},"AuthenticatedUserTenantDomain" : %{QUOTEDSTRING:tenant},"ServiceProviderName" : %{QUOTEDSTRING:serviceprovider},"RequestType" : %{QUOTEDSTRING:request},"RelyingParty" : %{QUOTEDSTRING:relyparty},"AuthenticatedIdPs" : %{QUOTEDSTRING:authenticatedidps} } \| Result : %{WORD:result}
FInally here is one version of the filter I am attempting:
if [type] == "wso2audit" {
grok {
match => "message" => ["%{TIMESTAMP_ISO8601:date}\]\s\sINFO {AUDIT_LOG}-\s\sInitiator : %{WORD:user}@carbon.super \| Action : %{WORD:action} \| Target : %{WORD:framework} \| Data : { "ContextIdentifier" : %{QUOTEDSTRING:context},"AuthenticatedUser" : %{QUOTEDSTRING:fulluser},"AuthenticatedUserTenantDomain" : %{QUOTEDSTRING:tenant},"ServiceProviderName" : %{QUOTEDSTRING:serviceprovider},"RequestType" : %{QUOTEDSTRING:request},"RelyingParty" : %{QUOTEDSTRING:relyparty},"AuthenticatedIdPs" : %{QUOTEDSTRING:authenticatedidps} } \| Result : %{WORD:result}"}]
remove_field => "framework"
remove_field => "tenant"
remove_field => "relyparty"
remove_field => "authenticatedips"
}
}
As you can see I am only interested in parsing out few of the fields and want to ignore several. Appreciate any advice as I am losing patience.