# Logstash: Optional fields in grok

**URL:** <https://discuss.elastic.co/t/logstash-optional-fields-in-grok/320604>\
**Category:** Logstash\
**Created:** [December 6, 2022, 3:46pm UTC](https://discuss.elastic.co/t/logstash-optional-fields-in-grok/320604 "2022-12-06T15:46:15Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![anon99430464](https://avatars.discourse-cdn.com/v4/letter/a/a587f6/32.png) [@anon99430464](https://discuss.elastic.co/u/anon99430464)\
**Post date:** [December 6, 2022, 3:46pm UTC](https://discuss.elastic.co/t/logstash-optional-fields-in-grok/320604/1 "2022-12-06T15:46:15Z")

</div>

Hi there

I'm trying to create a logstash for our logs. Our log files can look like the following

```auto
2022-11-22 10:43:59,061 INFO [SERVER1.Subscription] |>> | [ctx:0ecaa086-f9b7-4d0e-bce4-1b8513238745] [req:facdaf2c-b8e6-457a-9629-ddd777a011aa] [trx:200a9a69-7533-48d5-87cf-9b0c99178c41] [usr:-] [app:-] [clt:-] [que:QA.TST.1] --- { random json data }
2022-11-30 11:53:58,234 INFO [SERVER1.Subscription] | <=| [ctx:1184d2dc-4375-4760-82b3-2c88bbc525ff] [req:e1f7d5be-ba56-4ed0-a3e3-944c61ae6610] [trx:68c46c02-7362-42bd-a5a8-ddfd71894e36] [usr:-] [app:-] [srv:SERVER1] [sts:500] [mth:POST] [url:https://localhost:8080/post] --- { random json data }

```

You can see here that some fields are sometimes logged and other times they are not. (Like que, srv, sts, mth..) I'm trying to create a grok filter to read this data, but I'm having issues with square brackets I think.

I can get the pattern to work up until [app:] field, after which the fields can be different.

```auto
%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level}%{SPACE}\[%{GREEDYDATA:Type}\] \|%{GREEDYDATA}\| \[ctx:%{GREEDYDATA:ctx}\] \[req:%{GREEDYDATA:req}\] \[trx:%{GREEDYDATA:trx}\] \[usr:%{GREEDYDATA}\] --- %{GREEDYDATA:Json}

```

I've tried with the optional parameter `(\[que:?%{GREEDYDATA:que}])?` but it doesn't work with the second log file. I get the error:

> Provided Grok patterns do not match data in the input

I have also tried to add the optional parameter (?) after every character` (\[?q?u?e?:?%{GREEDYDATA:que}])?` but then it just starts using data from the next available field, which I don't want either. And if I add more fields like this, it eventually stops working with the same error as before (not matched).

> {  
> "que": "rl:[https://localhost:8080/post](https://localhost:8080/post)",  
> "level": "INFO",  
> "ctx": "1184d2dc-4375-4760-82b3-2c88bbc525ff",  
> "type": "SERVER1.Subscription",  
> "json\_data": "{ random json data }",  
> "trx": "68c46c02-7362-42bd-a5a8-ddfd71894e36",  
> "timestamp": "2022-11-30 11:53:58,234",  
> "req": "e1f7d5be-ba56-4ed0-a3e3-944c61ae6610"  
> }

Any ideas here on how to solve this?

Thank you!

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [December 6, 2022, 5:42pm UTC](https://discuss.elastic.co/t/logstash-optional-fields-in-grok/320604/2 "2022-12-06T17:42:57Z")

</div>

> [@anon99430464](#):
>
> Any ideas here on how to solve this?

Try using a kv filter

```
    grok { match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level}%{SPACE}\[%{GREEDYDATA:Type}\] \|%{GREEDYDATA}\| \[%{GREEDYDATA:kvData}\] --- %{GREEDYDATA:Json}" } }
    kv { source => "kvData" field_split_pattern => "\] \[" value_split => ":" }

```

will produce

```
       "app" => "-",
       "trx" => "68c46c02-7362-42bd-a5a8-ddfd71894e36",
      "Json" => "{ random json data }",
       "usr" => "-",
       "srv" => "SERVER1",
       "url" => "https://localhost:8080/post",
      "Type" => "SERVER1.Subscription",
       "mth" => "POST",
     "level" => "INFO",
       "sts" => "500",
       "req" => "e1f7d5be-ba56-4ed0-a3e3-944c61ae6610",
       "ctx" => "1184d2dc-4375-4760-82b3-2c88bbc525ff"

```

---

<div class="post-metadata">

**Author:** ![anon99430464](https://avatars.discourse-cdn.com/v4/letter/a/a587f6/32.png) [@anon99430464](https://discuss.elastic.co/u/anon99430464)\
**Post date:** [December 7, 2022, 2:35pm UTC](https://discuss.elastic.co/t/logstash-optional-fields-in-grok/320604/3 "2022-12-07T14:35:34Z")

</div>

That's perfect, thank you! My only issue currently is that the JSON property is full of escaped values.  
Is there a way not to escape the characters in a field? I also can't always convert it to JSON with the JSON filter, because it's not always JSON. Sometimes it's also a text message.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 4, 2023, 2:35pm UTC](https://discuss.elastic.co/t/logstash-optional-fields-in-grok/320604/4 "2023-01-04T14:35:45Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.