Multiline log (array) with logstash


(tikejhya) #1

I want to parse logfile with logstash which contains both single line and multiple line. [e.g first 2 lines with 1 line log entry whereas 3rd one has multiple line entry ]

ERROR - 2015-12-05 20:48:53 --> Could not find page
ERROR - 2015-12-05 20:48:53 --> Could not find VAR
ERROR - 2015-12-05 20:48:59 --> Array
(
[id] => 12344
[studentid] => 33
[fname] =>
[lname] =>
[address] => tokyo
)

This log entry is forwarded from client (logstatsh-forwarder) which sets type as "multilineclient"

filter{
if [type] == "multilineclient" {
multiline {
pattern => "^ERROR"
what => "previous"
}
grok{
match => {"message" => "%{LOGLEVEL:loglevel}\s+%{TIMESTAMP_ISO8601:timestamp}\s+%{DATA:message}({({[^}]+},?\s*)})?\s$(?(?m:.*))?"}
}

mutate {
        remove => [ "@loglevel" ]
}

}
}

I did try both Grok Debugger and grok constructer (but couldn't quite solve issue with LOGLEVEL being start of logfile ),

My multiline logs (array) are parsed as separate message.

message: [id] =>
message: [studentid] =>
message: [fname] =>
I was expecting this to come as single "message:"

Any suggestion?


(system) #2