How to split json value in log file using grok/regular expression

Elastic Stack Logstash
1

I have one log file which I need to extract the json content from the file and I need to parse it using logstash json filter. I wrote one grok pattern but which it is not working properly. Below is my log file.

2016-12-18 12:13:52.313 -08:00 [Information] 636176600323139749 1b2c4c40-3da6-46ff-b93f-0eb07a57f2a3 18 - API: GET AAA Sorry Page eq '9'&$expand=org($filter=org eq '0')
{
"Id": "1b",
"App": "D",
"User": "",
"Machine": "DC",
"RequestIpAddress": "xx.xxx.xxx",
"RequestHeaders": {
"Cache-Control": "no-transform",
"Connection": "close",
"Accept": "application/json"
},
"RequestTimestamp": "2016-12-18T12:13:52.2609587-08:00",
"ResponseContentType": "application/json",
"ResponseContentBody": {
"@od","value":[
{
"uid":"","sId":"10,org":[
{
"startDate":"2015-02-27T08:00:00Z","Code":"0","emailId":"xx@gg.COM"
}
]
}
]
},
"ResponseStatusCode": 200,
"ResponseHeaders": {
"Content-Type": "application/json;"
},
"ResponseTimestamp": "2016-12-18T12:13:52.3119655-08:00"
}

My Grok pattern

grok {
match => [ "message","%{TIMESTAMP_ISO8601:exclude}%{GREEDYDATA:exclude1}(?[\s])(?<json_value>[\W\w]+)"]
}

2

so your logfile contains 1 line of text and then a multiline json document ?
If so you need to use the multiline codec in the file input to collapse all those lines into one first.

Also, if you control the application that is logging, I'd suggest encoding a better log line that has a clear separation between the header and the json, or even logging everything as a json document
For example, putting:
2016-12-18 12:13:52.313 -08:00 into timestamp key
636176600323139749 1b2c4c40-3da6-46ff-b93f-0eb07a57f2a3 18 - API: GET https://aaa.com/o/v/S?$filter=uid eq '9'&$expand=org($filter=org eq '0') as log_header key

3

I used multiline codec and I am sending the full event as single message

4

The grok pattern which I wrote is not separating the json content..

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.