with the following pattern i got no grokfailure, but the message is not fully parsed-
...
if i try to parse the rest of the message it fails.
Works for me.
$ cat test.config
input { stdin { } }
output { stdout { codec => rubydebug } }
filter {
grok {
match => [
"message",
"%{DATA:application}!%{DATA:service}!%{DATA:id}!%{DATA:log_timestamp}:: %{DATA:level_info} %{DATA:level_log}: %{GREEDYDATA:msg}"
]
}
}
$ cat data
library!WindowsService_20!37d8!11/06/2017-11:00:13:: i INFO: Cleaned 0 batch records, 0 policies, 0 sessions, 0 cache entries, 0 snapshots, 0 chunks, 0 running jobs, 0 persisted streams, 0 segments, 0 segment mappings, 0 edit sessions.
$ /opt/logstash/bin/logstash -f test.config < data
Settings: Default pipeline workers: 8
Pipeline main started
{
"message" => "library!WindowsService_20!37d8!11/06/2017-11:00:13:: i INFO: Cleaned 0 batch records, 0 policies, 0 sessions, 0 cache entries, 0 snapshots, 0 chunks, 0 running jobs, 0 persisted streams, 0 segments, 0 segment mappings, 0 edit sessions.",
"@version" => "1",
"@timestamp" => "2017-11-22T08:46:14.353Z",
"host" => "lnxolofon",
"application" => "library",
"service" => "WindowsService_20",
"id" => "37d8",
"log_timestamp" => "11/06/2017-11:00:13",
"level_info" => "i",
"level_log" => "INFO",
"msg" => "Cleaned 0 batch records, 0 policies, 0 sessions, 0 cache entries, 0 snapshots, 0 chunks, 0 running jobs, 0 persisted streams, 0 segments, 0 segment mappings, 0 edit sessions."
}
Pipeline main has been shutdown
stopping pipeline {:id=>"main"}
What did you mean more exakt pattern ?
The DATA and GREEDYDATA patterns match any characters, including your delimiter characters. Under some circumstances this can result in incorrect matches. In this case you could use (?<application>[^!]+) to matches everything up to (but not including) the next exclamation point. This avoids incorrect matches.
Secondly, excessive use of DATA and GREEDYDATA can result in extremely bad performance since the input string can be parsed in multiple ways and the regexp engine needs to go over the whole string multiple times. For the same reason you should start your expression with ^.
