I want to parse logfile with logstash which contains both single line and multiple line. [e.g first 2 lines with 1 line log entry whereas 3rd one has multiple line entry ]
ERROR - 2015-12-05 20:48:53 --> Could not find page
ERROR - 2015-12-05 20:48:53 --> Could not find VAR
ERROR - 2015-12-05 20:48:59 --> Array
(
[id] => 12344
[studentid] => 33
[fname] =>
[lname] =>
[address] => tokyo
)
This log entry is forwarded from client (logstatsh-forwarder) which sets type as "multilineclient"
filter{
if [type] == "multilineclient" {
multiline {
pattern => "^ERROR"
what => "previous"
}
grok{
match => {"message" => "%{LOGLEVEL:loglevel}\s+%{TIMESTAMP_ISO8601:timestamp}\s+%{DATA:message}({({[^}]+},?\s*)})?\s$(?(?m:.*))?"}
}
mutate { remove => [ "@loglevel" ] }}
}
I did try both Grok Debugger and grok constructer (but couldn't quite solve issue with LOGLEVEL being start of logfile ),
My multiline logs (array) are parsed as separate message.
message: [id] =>
message: [studentid] =>
message: [fname] =>
I was expecting this to come as single "message:"
Any suggestion?