Hi there
I'm trying to create a logstash for our logs. Our log files can look like the following
2022-11-22 10:43:59,061 INFO [SERVER1.Subscription] |>> | [ctx:0ecaa086-f9b7-4d0e-bce4-1b8513238745] [req:facdaf2c-b8e6-457a-9629-ddd777a011aa] [trx:200a9a69-7533-48d5-87cf-9b0c99178c41] [usr:-] [app:-] [clt:-] [que:QA.TST.1] --- { random json data }
2022-11-30 11:53:58,234 INFO [SERVER1.Subscription] | <=| [ctx:1184d2dc-4375-4760-82b3-2c88bbc525ff] [req:e1f7d5be-ba56-4ed0-a3e3-944c61ae6610] [trx:68c46c02-7362-42bd-a5a8-ddfd71894e36] [usr:-] [app:-] [srv:SERVER1] [sts:500] [mth:POST] [url:https://localhost:8080/post] --- { random json data }
You can see here that some fields are sometimes logged and other times they are not. (Like que, srv, sts, mth..) I'm trying to create a grok filter to read this data, but I'm having issues with square brackets I think.
I can get the pattern to work up until [app:] field, after which the fields can be different.
%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level}%{SPACE}\[%{GREEDYDATA:Type}\] \|%{GREEDYDATA}\| \[ctx:%{GREEDYDATA:ctx}\] \[req:%{GREEDYDATA:req}\] \[trx:%{GREEDYDATA:trx}\] \[usr:%{GREEDYDATA}\] --- %{GREEDYDATA:Json}
I've tried with the optional parameter (\[que:?%{GREEDYDATA:que}])?
but it doesn't work with the second log file. I get the error:
Provided Grok patterns do not match data in the input
I have also tried to add the optional parameter (?) after every character (\[?q?u?e?:?%{GREEDYDATA:que}])?
but then it just starts using data from the next available field, which I don't want either. And if I add more fields like this, it eventually stops working with the same error as before (not matched).
{
"que": "rl:https://localhost:8080/post",
"level": "INFO",
"ctx": "1184d2dc-4375-4760-82b3-2c88bbc525ff",
"type": "SERVER1.Subscription",
"json_data": "{ random json data }",
"trx": "68c46c02-7362-42bd-a5a8-ddfd71894e36",
"timestamp": "2022-11-30 11:53:58,234",
"req": "e1f7d5be-ba56-4ed0-a3e3-944c61ae6610"
}
Any ideas here on how to solve this?
Thank you!