Logstash: Optional fields in grok

[Schoof]

Hi there

I'm trying to create a logstash for our logs. Our log files can look like the following

2022-11-22 10:43:59,061 INFO  [SERVER1.Subscription] |>>  | [ctx:0ecaa086-f9b7-4d0e-bce4-1b8513238745] [req:facdaf2c-b8e6-457a-9629-ddd777a011aa] [trx:200a9a69-7533-48d5-87cf-9b0c99178c41] [usr:-] [app:-] [clt:-] [que:QA.TST.1]  --- { random json data }
2022-11-30 11:53:58,234 INFO  [SERVER1.Subscription] |  <=| [ctx:1184d2dc-4375-4760-82b3-2c88bbc525ff] [req:e1f7d5be-ba56-4ed0-a3e3-944c61ae6610] [trx:68c46c02-7362-42bd-a5a8-ddfd71894e36] [usr:-] [app:-] [srv:SERVER1] [sts:500] [mth:POST] [url:https://localhost:8080/post]  --- { random json data }

You can see here that some fields are sometimes logged and other times they are not. (Like que, srv, sts, mth..) I'm trying to create a grok filter to read this data, but I'm having issues with square brackets I think.

I can get the pattern to work up until [app:] field, after which the fields can be different.

%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level}%{SPACE}\[%{GREEDYDATA:Type}\] \|%{GREEDYDATA}\| \[ctx:%{GREEDYDATA:ctx}\] \[req:%{GREEDYDATA:req}\] \[trx:%{GREEDYDATA:trx}\] \[usr:%{GREEDYDATA}\]  --- %{GREEDYDATA:Json}

I've tried with the optional parameter (\[que:?%{GREEDYDATA:que}])? but it doesn't work with the second log file. I get the error:

Provided Grok patterns do not match data in the input

I have also tried to add the optional parameter (?) after every character (\[?q?u?e?:?%{GREEDYDATA:que}])? but then it just starts using data from the next available field, which I don't want either. And if I add more fields like this, it eventually stops working with the same error as before (not matched).

{
"que": "rl:https://localhost:8080/post",
"level": "INFO",
"ctx": "1184d2dc-4375-4760-82b3-2c88bbc525ff",
"type": "SERVER1.Subscription",
"json_data": "{ random json data }",
"trx": "68c46c02-7362-42bd-a5a8-ddfd71894e36",
"timestamp": "2022-11-30 11:53:58,234",
"req": "e1f7d5be-ba56-4ed0-a3e3-944c61ae6610"
}

Any ideas here on how to solve this?

Thank you!

[Badger]

@Schoof

Any ideas here on how to solve this?

Try using a kv filter

    grok { match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level}%{SPACE}\[%{GREEDYDATA:Type}\] \|%{GREEDYDATA}\| \[%{GREEDYDATA:kvData}\]  --- %{GREEDYDATA:Json}" } }
    kv { source => "kvData" field_split_pattern => "\] \[" value_split => ":" }

will produce

       "app" => "-",
       "trx" => "68c46c02-7362-42bd-a5a8-ddfd71894e36",
      "Json" => "{ random json data }",
       "usr" => "-",
       "srv" => "SERVER1",
       "url" => "https://localhost:8080/post",
      "Type" => "SERVER1.Subscription",
       "mth" => "POST",
     "level" => "INFO",
       "sts" => "500",
       "req" => "e1f7d5be-ba56-4ed0-a3e3-944c61ae6610",
       "ctx" => "1184d2dc-4375-4760-82b3-2c88bbc525ff"

[Schoof]

That's perfect, thank you! My only issue currently is that the JSON property is full of escaped values.
Is there a way not to escape the characters in a field? I also can't always convert it to JSON with the JSON filter, because it's not always JSON. Sometimes it's also a text message.