# Grok expression works, but causes logstash config check to fail. Special characters in log an issue?

**URL:** <https://discuss.elastic.co/t/grok-expression-works-but-causes-logstash-config-check-to-fail-special-characters-in-log-an-issue/156107>\
**Category:** Logstash\
**Created:** [November 9, 2018, 8:48pm UTC](https://discuss.elastic.co/t/grok-expression-works-but-causes-logstash-config-check-to-fail-special-characters-in-log-an-issue/156107 "2018-11-09T20:48:17Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![hueyg](https://avatars.discourse-cdn.com/v4/letter/h/8c91f0/32.png) [@hueyg](https://discuss.elastic.co/u/hueyg)\
**Post date:** [November 9, 2018, 8:48pm UTC](https://discuss.elastic.co/t/grok-expression-works-but-causes-logstash-config-check-to-fail-special-characters-in-log-an-issue/156107/1 "2018-11-09T20:48:17Z")

</div>

I have a grok expression for an application where the log contains square brackets, curly brackets, and pipe characters. The expression works in the grok debugger but fails when I use it in the logstash filter.

Here is a sample of the data:

```
[2018-11-09 10:30:25,698] INFO {AUDIT_LOG}- Initiator : J00168813@carbon.super | Action : Login | Target : ApplicationAuthenticationFramework | Data : { "ContextIdentifier" : "0010e177-ebfa-43a6-9b8c-28ed64370687","AuthenticatedUser" : "J00168813@carbon.super","AuthenticatedUserTenantDomain" : "null","ServiceProviderName" : "BanAdmin_PROD","RequestType" : "cassso","RelyingParty" : "BanAdmin_PROD","AuthenticatedIdPs" : "eyJ0eXAiOiJKV1QiLCAiYWxnIjoibm9uZSJ9.eyJpc3MiOiJ3c28yIiwiZXhwIjoxNTQxNzgxMDI1NjgwMzAwMCwiaWF0IjoxNTQxNzgxMDI1NjgwLCJpZHBzIjpbeyJpZHAiOiJMT0NBTCIsImF1dGhlbnRpY2F0b3IiOiJCYXNpY0F1dGhlbnRpY2F0b3IifV19." } | Result : Success

```

Here is my grok expression. Notice the special characters. I have another log file with square brackets which works fine. I know I have a mix of escape characters but this is after way too many tries so forgive my inconsistent attempts:

```
		%{TIMESTAMP_ISO8601:date}\]\s\sINFO {AUDIT_LOG}-\s\sInitiator : %{WORD:user}@carbon.super \| Action : %{WORD:action} \| Target : %{WORD:framework} \| Data : { "ContextIdentifier" : %{QUOTEDSTRING:context},"AuthenticatedUser" : %{QUOTEDSTRING:fulluser},"AuthenticatedUserTenantDomain" : %{QUOTEDSTRING:tenant},"ServiceProviderName" : %{QUOTEDSTRING:serviceprovider},"RequestType" : %{QUOTEDSTRING:request},"RelyingParty" : %{QUOTEDSTRING:relyparty},"AuthenticatedIdPs" : %{QUOTEDSTRING:authenticatedidps} } \| Result : %{WORD:result}

```

FInally here is one version of the filter I am attempting:

```
 if [type] == "wso2audit" {
grok {
  match => "message" => ["%{TIMESTAMP_ISO8601:date}\]\s\sINFO {AUDIT_LOG}-\s\sInitiator : %{WORD:user}@carbon.super \| Action : %{WORD:action} \| Target : %{WORD:framework} \| Data : { "ContextIdentifier" : %{QUOTEDSTRING:context},"AuthenticatedUser" : %{QUOTEDSTRING:fulluser},"AuthenticatedUserTenantDomain" : %{QUOTEDSTRING:tenant},"ServiceProviderName" : %{QUOTEDSTRING:serviceprovider},"RequestType" : %{QUOTEDSTRING:request},"RelyingParty" : %{QUOTEDSTRING:relyparty},"AuthenticatedIdPs" : %{QUOTEDSTRING:authenticatedidps} } \| Result : %{WORD:result}"}]
  remove_field => "framework"
  remove_field => "tenant"
  remove_field => "relyparty"
  remove_field => "authenticatedips"
}

```

}

As you can see I am only interested in parsing out few of the fields and want to ignore several. Appreciate any advice as I am losing patience.

---

<div class="post-metadata">

**Author:** ![Makra](https://avatars.discourse-cdn.com/v4/letter/m/8491ac/32.png) [@Makra](https://discuss.elastic.co/u/Makra)\
**Post date:** [November 10, 2018, 3:51am UTC](https://discuss.elastic.co/t/grok-expression-works-but-causes-logstash-config-check-to-fail-special-characters-in-log-an-issue/156107/2 "2018-11-10T03:51:01Z")

</div>

Hi

@hueyg

The GROK is okay, you just need to escape the double quotes for Logstash to work. The following worked in my logstash instance after escaping the double quotes inside grok.

`match => ["message" , "%{TIMESTAMP_ISO8601:date}\]\s\sINFO {AUDIT_LOG}-\s\sInitiator : %{WORD:user}@carbon.super \| Action : %{WORD:action} \| Target : %{WORD:framework} \| Data : { \"ContextIdentifier\" : %{QUOTEDSTRING:context},\"AuthenticatedUser\" : %{QUOTEDSTRING:fulluser},\"AuthenticatedUserTenantDomain\" : %{QUOTEDSTRING:tenant},\"ServiceProviderName\" : %{QUOTEDSTRING:serviceprovider},\"RequestType\" : %{QUOTEDSTRING:request},\"RelyingParty\" : %{QUOTEDSTRING:relyparty},\"AuthenticatedIdPs\" : %{QUOTEDSTRING:authenticatedidps} } \| Result : %{WORD:result}"]`

[[Album] Imgur: The magic of the Internet ![](https://i.imgur.com/AypQJE3.jpg?fb "            
    

    Imgur: The magic of the Internet") ](https://imgur.com/a/UfFZOAe)

Also use mutate filter to remove the fields.

```
mutate {
         remove_field => ["framework","tenant","relyparty","authenticatedips"]
       }
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 8, 2018, 3:51am UTC](https://discuss.elastic.co/t/grok-expression-works-but-causes-logstash-config-check-to-fail-special-characters-in-log-an-issue/156107/3 "2018-12-08T03:51:06Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.