I have the conf file as below:
input {
file {
path => "E:/Logs/A_logs.log"
start_position => "beginning"
codec => multiline {
pattern => "^\[%{TIMESTAMP_ISO8601}\] "
negate => true
what => "previous"
}
type => "test" # a type to identify those logs (will need this later)
sincedb_path => "NUL"
ignore_older => 0
}
}
filter {
if [type] == "test"{
if [path] =~ "A"{
mutate { replace => {type => "A"}}
if "multiline" in [tags] {
grok {
match => ["message", "%{JAVASTACKTRACEPART}"]
}
}
mutate {
gsub => ['message', "\n", " "]
}
mutate {
gsub => ['message', "\r", " "]
}
grok {
match => {"message" => "[%{TIMESTAMP_ISO8601:TIMESTAMP}]%{SPACE}%{WORD:loglevel}%{SPACE}%{WORD:esb_info}%{SPACE}%{GREEDYDATA:esb_info1}(?(.|\r|\n)*)"}
}
}
}
}
output {
if "_grokparsefailure" not in [tags] {
elasticsearch {
hosts => ["localhost:9200"]
sniffing => true
manage_template => false
index => "%{[type]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
}
And my pattern as specified above goes like:
\[%{TIMESTAMP_ISO8601:TIMESTAMP}\]%{SPACE}%{WORD:loglevel}%{SPACE}%{WORD:esb_info}%{SPACE}%{GREEDYDATA:esb_info1}(?(.|\r|\n)*)
The logs have format as below :
[2017-10-10 00:00:25,091] DEBUG EsbSSSEvents18 a.abc.sample.mgmt.service.impl.CheckRR17Impl - Processing incoming XMLEvent:
<pfx4:CheckQuery xmlns:pfx4="https://blablah" xmlns:SOAP-ENV="http://hhhh">
pfx4:Header
pfx4:sss1111</pfx4:sss>
pfx4:ddd2017-10-10T00:00:15+02:00</pfx4:ddd>
pfx4:actionUpdate</pfx4:action>
pfx4:bysytem</pfx4:by>
</pfx4:Header>
[2017-10-10 00:00:25,092] DEBUG EsbSSSEvents18 a.abc.sample.mgmt.service.impl.CheckRR17Impl - Generated Events:
1: Event: Type = Commit, SSS = 122344 Timestamp = Tue Oct 10 00:00:15 CEST 2017, Action = UPDATE, Date = Tue Oct 09 00:00:00 CEST 2018
[2017-10-10 00:00:25,092] DEBUG EsbSSSEvents18 a.abc.sample.mgmt.service.impl.CheckRR17ImplControl - Mapping Date for 3333 to UpdaterClass
[2017-10-10 00:00:25,091] DEBUG EsbSSSEvents18 a.abc.sample.mgmt.service.impl.CheckRR17Impl - Processing incoming XMLEvent:
<pfx4:Pass xmlns:pfx4="https://blablah" xmlns:SOAP-ENV="http://hhhh">
pfx4:Header
pfx4:sss222</pfx4:sss>
pfx4:ddd2017-10-10T00:00:15+02:00</pfx4:ddd>
pfx4:actionCLOSE</pfx4:action>
pfx4:byqqqq</pfx4:by>
</pfx4:Header>
The message is parsed only for first line of log and it puts the entire log in field "stack" and does not break on second occurrence of date. Error is as below:
"type" => "A",
"tags" => [
[0] "multiline",
[1] "_grokparsefailure"
Can anyone please tell what is going wrong
