zozo6015
(Peter)
September 19, 2019, 8:26am
1
Hello,
I have a long used grok pattern match that stopped working since upgraded to ELK 7.3.0.
grok {
match => { "message" => ["%{MONTH:month} %{MONTHDAY:day}, %{YEAR:year} %{TIME:time} %{WORD:day_period} %{NOTSPACE:[system][jetty][class]} %{WORD:[system][jetty][method]}\\n%{GREEDYMULTILINE:multiline}",
"%{WORD:severity} in thread \"%{THREADNAME:threadName}\" %{GREEDYDATA:[system][jetty][data]}",
"%{WORD:severity} in thread \"%{THREADNAME:threadName}\" %{NOTSPACE:[system][jetty][exceptionClass]}: %{GREEDYDATA:exceptionMessage}\\n %{GREEDYMULTILINE:exceptionMultiline}"
] }
pattern_definitions => {
"GREEDYMULTILINE" => "(.|\r|\n)*",
"THREADNAME" => "[^\"]+"
}
remove_field => "host"
}
This grok match does not split the message on fields while using grok pattern matcher from kibana all goes well as it can be seen in the picture uploaded.
Any suggestion how to fix it?
Do you have a correctly configured multiline codec in place?
zozo6015
(Peter)
September 23, 2019, 5:59pm
4
The codec was working for months on 6.x. Suddenly stopped working when I switched to 7.3.0.
Then it seems to be the codec that is the issue. How is it configured?
zozo6015
(Peter)
September 23, 2019, 6:03pm
6
Not sure which codec are we talking about? This is a grok matching filter option which has the possibility to create custom patterns.
Which inputs are you using? Are the events that are not parsed correctly having multiple lines in the message field?
zozo6015
(Peter)
September 23, 2019, 6:10pm
8
I am using beats the information is coming from filebeat. The filebeat is configured correctly since the full message is correct. The issue is with the grok custom pattern feature. For some reason the same pattern is splitting the message into fields but when it gets to the grok filter it does not work.
I am not sure I follow. Can you show anevent that has been indexed and not been parsed correctly?
zozo6015
(Peter)
September 23, 2019, 6:15pm
10
Here you have a link to an event that is not splitted into fields.
https://pastebin.com/z1LRBQPx
I think it might be the case of the message not interpretting the \n
while the grok considers that as a word and not a new line.
Badger
September 23, 2019, 7:06pm
11
You cannot have a comma between patterns. It will not compile.
If you have a literal newline in your data then use one in the pattern
"%{MONTH:month} %{MONTHDAY:day}, %{YEAR:year} %{TIME:time} %{WORD:day_period} %{NOTSPACE:[system][jetty][class]} %{WORD:[system][jetty][method]}
%{GREEDYMULTILINE:multiline}"
zozo6015
(Peter)
September 23, 2019, 7:12pm
12
Changed the config but still not splitting up the message fields on smaller fields.
grok {
match => { "message" => ["%{MONTH:month} %{MONTHDAY:day}, %{YEAR:year} %{TIME:time} %{WORD:day_period} %{NOTSPACE:[system][jetty][class]} %{WORD:[system][jetty][method]}(:?\n|\\n)%{GREEDYMULTILINE:multiline}",
"%{WORD:severity} in thread \"%{THREADNAME:threadName}\" %{GREEDYDATA:[system][jetty][data]}",
"%{WORD:severity} in thread \"%{THREADNAME:threadName}\" %{NOTSPACE:[system][jetty][exceptionClass]}: %{GREEDYDATA:exceptionMessage}(:?\n|\\n) %{GREEDYMULTILINE:exceptionMultiline}"
] }
pattern_definitions => {
"GREEDYMULTILINE" => "(.|\r|\n)*"
"THREADNAME" => "[^\"]+"
}
remove_field => "host"
}
system
(system)
Closed
October 21, 2019, 7:12pm
13
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.