Hi,
I'm new to logstash and I'm having trouble parsing my logfile. Logfile (sanitized) can be found here:
http://pastebin.com/6nsSVMQj
There are three different types (outbound, inbound and "misc") of log entries in the same file, so I grok three different filters against the same message. My patterns are fine and when I change the order of entries in the logfile it still parses ok (but only the first entry)
My complete config looks like this:
input {
file{
path => "/home/steven/sip.log.2"
start_position => beginning
# logstash stores the lastrun=> so we trick it
sincedb_path => "/dev/null"
#if logentry does not start with date it's part of previous entry
codec => multiline {
pattern => "^\[%{TIMESTAMP_ISO8601:logdate} \]"
negate => "true"
what => "previous"
}
}
}
filter {
grok {
break_on_match => true
match => {"message" => [
"\[%{TIMESTAMP_ISO8601:logdate} \] %{LOGLEVEL:level} AbstractLoggingInterceptor: Inbound Message\n----------------------------\n%{GREEDYDATA:id}\n%{GREEDYDATA:responsecode}\n%{GREEDYDATA:encoding}\n%{GREEDYDATA:contenttype}\n%{GREEDYDATA:headers}\n%{GREEDYDATA:payload}\n--------------------------------------"
,"\[%{TIMESTAMP_ISO8601:logdate} \] %{LOGLEVEL:level} AbstractLoggingInterceptor: Outbound Message\n---------------------------\n%{GREEDYDATA:id}\n%{GREEDYDATA:responsecode}\n%{GREEDYDATA:encoding}\n%{GREEDYDATA:contenttype}\n%{GREEDYDATA:headers}\n%{GREEDYDATA:payload}\n--------------------------------------"
,"\[%{TIMESTAMP_ISO8601:logdate} \] %{LOGLEVEL:level} %{GREEDYDATA:type}"]}
}
}
output {
elasticsearch {}
# stdout {
# codec => "json"
# }
}
This is the result that I receive in elasticsearch. As you can see it's only the first entry in the logfile:
`[steven@localhost ~]$ curl -XPOST 'http://localhost:9200/logstash-2016.01.19/_search?q=*&pretty'
{
"took" : 20,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 1.0,
"hits" : [ {
"_index" : "logstash-2016.01.19",
"_type" : "logs",
"_id" : "AVJZ_BxC0rT1ILfRgQQB",
"_score" : 1.0,
"_source":{"@timestamp":"2016-01-19T13:03:58.412Z","message":"[2015-12-17 08:20:29,028 ] INFO AbstractLoggingInterceptor: Outbound Message\n---------------------------\nID: 4911888\nAddress: https://baswebapp1:343/svc/Authentication.svc\nEncoding: UTF-8\nContent-Type: application/soap+xml; action=\"http://asp.net/ApplicationServices/v200/AuthenticationService/Login\"\nHeaders: {Accept=[*/*]}\nPayload: <test outbound>\n--------------------------------------","@version":"1","tags":["multiline"],"host":"localhost.localdomain","path":"/home/steven/sip.log.2","logdate":"2015-12-17 08:20:29,028","level":"INFO","id":"ID: 4911888","responsecode":"Address: https://baswebapp1:343/svc/Authentication.svc","encoding":"Encoding: UTF-8","contenttype":"Content-Type: application/soap+xml; action=\"http://asp.net/ApplicationServices/v200/AuthenticationService/Login\"","headers":"Headers: {Accept=[*/*]}","payload":"Payload: <test outbound>"}
} ]
}
}
`
What am I doing wrong?