# Correct Grok Pattern for an optional field including spaces

**URL:** <https://discuss.elastic.co/t/correct-grok-pattern-for-an-optional-field-including-spaces/188571>\
**Category:** Logstash\
**Created:** [July 2, 2019, 7:10pm UTC](https://discuss.elastic.co/t/correct-grok-pattern-for-an-optional-field-including-spaces/188571 "2019-07-02T19:10:08Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![sandeepkanabar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sandeepkanabar/32/79399_2.png) [@sandeepkanabar](https://discuss.elastic.co/u/sandeepkanabar)\
**Post date:** [July 2, 2019, 7:10pm UTC](https://discuss.elastic.co/t/correct-grok-pattern-for-an-optional-field-including-spaces/188571/1 "2019-07-02T19:10:08Z")

</div>

In some of the log messages, an optional field `[opt_field=xyz]` or `[opt_field=]` is present. I arrived at the following pattern to handle it:

```
(?m)%{TIMESTAMP_ISO8601:Src_CreateDate} %{LOGLEVEL:Src_Severity} \[(?<Src_Thread>[^\]]+)\] (\s*\[opt_field=%{DATA:opt_field}\]\s+)?%{JAVACLASS:Src_ClassName} - \{%{GREEDYDATA:Src_LogMsg}\}

```

While the pattern works (tested in [grokconstructor](https://grokconstructor.appspot.com/do/match#result)), I wanted inputs on whether anything needs to be changed in it?

Here are the 4 different sample log messages with the conditions that need to be handled:

```
2019-07-01 11:20:35,539 INFO [Consumer-1] com.foo.webservices.es.handler.Logger - {msg={\"processTime\":1561980034949,\"fieldA\":\"AAAAAAAAAAAA\",\"fieldB\":\"-123456\",\"fieldC\":\"Value_C\",\"fieldD\":false}, errormsg=no record found}
2019-07-01 11:20:36,942 INFO [exec-31] [opt_field=] com.foo.webservices.es.handler.Logger - {abc=foobar, def=barfoo}
2019-07-01 11:20:35,664 INFO [Consumer-2] [opt_field=opt1_13d67663-615f-4689-9af1-3fa556c84067] com.foo.webservices.es.handler.Logger - {msg={\"processTime\":1561980034694,\"fieldA\":\"AAAAAAAAA\",\"fieldB\":\"-567890\"}, hid=host_id}
2019-07-01 11:20:35,664 INFO [Consumer-2][opt_field=opt1_13d67663-615f-4689-9af1-3fa556c84067] com.foo.webservices.es.handler.Logger - {msg={\"processTime\":1561980034694,\"fieldA\":\"AAAAAAAAA\",\"fieldB\":\"-567890\"}, hid=host_id}

```

`1st Msg`: Optional field is `not` present. Pattern should `match`.  
`2nd Msg`: Optional field `present` albeit with `null value`. Pattern should `match`.  
`3rd Msg`: Optional field is `present`. Pattern should `match`.  
`4th Msg`: Optional field is `present` but does `not` have a `space` after `thread_name`. Pattern should `not match`.

Can someone provide insights as to whether this pattern is correct and what could be done better, if anything.

ELK Stack Version: 5.5.1

Thank you.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 2, 2019, 7:28pm UTC](https://discuss.elastic.co/t/correct-grok-pattern-for-an-optional-field-including-spaces/188571/2 "2019-07-02T19:28:20Z")

</div>

With 7.2 I get the match / not match that you say you want. For [opt\_field=] it does match but no opt\_field field is added to the event. That is as expected.

---

<div class="post-metadata">

**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\
**Post date:** [July 3, 2019, 6:32pm UTC](https://discuss.elastic.co/t/correct-grok-pattern-for-an-optional-field-including-spaces/188571/3 "2019-07-03T18:32:37Z")

</div>

With 5.6.9 (the closest version I have to yours locally), I get the results that you say you want.

- In general, we recommend that grok patterns are _anchored_ (e.g., they begin with a _start-of-line_ anchor `^` or beginning-of-input anchor `\A`), which allows the pattern to give up faster when it doesn't find a match starting at the beginning; without anchors, a failed match will be attempted again starting with the 2nd character in the line, and again from the third character, etc.
- the pattern may be more performant if you used a more restricted pattern than `DATA`, as the pattern it expands to typically ends up capturing too much, requiring the parser to backtrack; for example, I assume that you would expect the value to not contain a closing square-bracket, so you could define one or more [pattern definitions](https://www.elastic.co/guide/en/logstash/5.6/plugins-filters-grok.html#plugins-filters-grok-pattern_definitions) as so:

```auto
  pattern_definitions => {
    "NOT_CLOSE_BRACKET" => "[^\]]*"
  }

```

OR, the pattern could be defined in pure regex, as you already do for the `Src_Thread` field:

```auto
(?m)%{TIMESTAMP_ISO8601:Src_CreateDate} %{LOGLEVEL:Src_Severity} \[(?<Src_Thread>[^\]]+)\] (\[opt_field=(?<opt_field>[^\]]*)\] )?%{JAVACLASS:Src_ClassName} - \{%{GREEDYDATA:Src_LogMsg}\}

```

---

<div class="post-metadata">

**Author:** ![sandeepkanabar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sandeepkanabar/32/79399_2.png) [@sandeepkanabar](https://discuss.elastic.co/u/sandeepkanabar)\
**Post date:** [July 3, 2019, 11:09pm UTC](https://discuss.elastic.co/t/correct-grok-pattern-for-an-optional-field-including-spaces/188571/4 "2019-07-03T23:09:33Z")

</div>

> [@yaauie](#):
>
> (?m)%{TIMESTAMP\_ISO8601:Src\_CreateDate} %{LOGLEVEL:Src\_Severity} [(?\<Src\_Thread\>[^]]+)] ([opt\_field=(?\<opt\_field\>[^]]\*)] )?%{JAVACLASS:Src\_ClassName} - {%{GREEDYDATA:Src\_LogMsg}}

Excellent suggestion Ry. Implemented it.

> [@yaauie](#):
>
> In general, we recommend that grok patterns are _anchored_ (e.g., they begin with a _start-of-line_ anchor `^` or beginning-of-input anchor `\A` )

You mean to say - Like this?

```
 ^(?m)%{TIMESTAMP_ISO8601:Src_CreateDate} %{LOGLEVEL:Src_Severity} \[(?<Src_Thread>[^\]]+)\] (\[opt_field=(?<opt_field>[^\]]*)\] )?%{JAVACLASS:Src_ClassName} - \{%{GREEDYDATA:Src_LogMsg}\}

```

Also, how do you colour the code 🙂 For e.g your few code syntax appears in red colour. Mind sharing?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 4, 2019, 1:41pm UTC](https://discuss.elastic.co/t/correct-grok-pattern-for-an-optional-field-including-spaces/188571/5 "2019-07-04T13:41:00Z")

</div>

If you use \</\> or indent text using four spaces it will appear like this

```
pattern_definitions => {
    "NOT_CLOSE_BRACKET" => "[^\]]*"
}

```

If you use three backticks on a line before and after it will appear like this

```auto
  pattern_definitions => {
    "NOT_CLOSE_BRACKET" => "[^\]]*"
  }

```

---

<div class="post-metadata">

**Author:** ![sandeepkanabar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sandeepkanabar/32/79399_2.png) [@sandeepkanabar](https://discuss.elastic.co/u/sandeepkanabar)\
**Post date:** [July 4, 2019, 3:18pm UTC](https://discuss.elastic.co/t/correct-grok-pattern-for-an-optional-field-including-spaces/188571/6 "2019-07-04T15:18:11Z")

</div>

> [@Badger](#):
>
> if you use three backticks on a line before and after it will appear like this

```auto
{ "Awesome" => "ThankYou @Badger " }

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 1, 2019, 3:18pm UTC](https://discuss.elastic.co/t/correct-grok-pattern-for-an-optional-field-including-spaces/188571/7 "2019-08-01T15:18:13Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.