Hi there,
So I have a custom application with a log file that have multiple type of lines, normal syslog and also json.
It looks like this:
2019-03-07T11:29:52+0000 | 10.0.0.0 | [3rdParty] xxxxxxxxxxxx || INFO | REQUEST | PUT https://10.0.0.0/xxxx | HEADERS: Accept-Encoding: gzip, deflate
M2M-Tracking-Token: xxxxxx
Content-Length: 2
Content-Type: application/json
Host: 10.0.0.0
User-Agent: Zend_Http_Client
Connection: Keep-Alive
M2M-Apikey: xxxxxxxx
X-Forwarded-Server: 10.0.0.0
X-Forwarded-Host: 10.0.0.0
X-Forwarded-Proto: https
X-Forwarded-For: 10.0.0.0
BODY: []
2019-03-07T11:29:52+0000 | 10.0.0.0 | [3rdParty] xxxxxxxxxxxx | xxxxxxx | INFO | RESPONSE | 200 OK PUT https://10.0.0.0 | 32ms | HEADERS: Content-Type: application/json
Content-Length: 473
X-CSRF-Token: None
X-Tracking-Token: xxxxxxx
BODY: {
"serviceSession": {
"xxxxxx",
"xxxxxx",
"xxxxxx",
"xxxxxx",
"xxxxxx",
"xxxxxx",
"xxxxxx",
"xxxxxx",
"testID: xxxxxx",
}
}
So I configured filebeat to use multiline since I want to "merge" them by timestamp date, otherwise I will get a lot of lines inside message and it`s hard to grok when you have a log with hundred of lines.
multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
multiline.negate: true
multiline.match: after
Now I get in my full length under message so I can play with grok to extract some relevant data.
I have some questions:
- Is there a way to mark that
BODY: {}
as json? - Is there a way to grok through this json part to extract
testID
field? a find function? - In the first part of the log I have a response time measured in milliseconds, can I extract that as a metric
%NUMBER
field?
Thanks for your time!