Grok pattern does not work in logstash however it works in kibana

[zozo6015 (Peter)]

Hello,

I have a long used grok pattern match that stopped working since upgraded to ELK 7.3.0.

grok {
   match => { "message" => ["%{MONTH:month} %{MONTHDAY:day}, %{YEAR:year} %{TIME:time} %{WORD:day_period} %{NOTSPACE:[system][jetty][class]} %{WORD:[system][jetty][method]}\\n%{GREEDYMULTILINE:multiline}",
                           "%{WORD:severity} in thread \"%{THREADNAME:threadName}\" %{GREEDYDATA:[system][jetty][data]}",
                           "%{WORD:severity} in thread \"%{THREADNAME:threadName}\" %{NOTSPACE:[system][jetty][exceptionClass]}: %{GREEDYDATA:exceptionMessage}\\n %{GREEDYMULTILINE:exceptionMultiline}"
                           ] }
   pattern_definitions => { 
      "GREEDYMULTILINE" => "(.|\r|\n)*",
      "THREADNAME" => "[^\"]+"
   }
   remove_field => "host"
}

This grok match does not split the message on fields while using grok pattern matcher from kibana all goes well as it can be seen in the picture uploaded.

Any suggestion how to fix it?

[zozo6015 (Peter)]

Any suggestion for this?

[Christian_Dahlqvist]

Do you have a correctly configured multiline codec in place?

[zozo6015 (Peter)]

The codec was working for months on 6.x. Suddenly stopped working when I switched to 7.3.0.

[Christian_Dahlqvist]

Then it seems to be the codec that is the issue. How is it configured?

[zozo6015 (Peter)]

Not sure which codec are we talking about? This is a grok matching filter option which has the possibility to create custom patterns.

[Christian_Dahlqvist]

Which inputs are you using? Are the events that are not parsed correctly having multiple lines in the message field?

[zozo6015 (Peter)]

I am using beats the information is coming from filebeat. The filebeat is configured correctly since the full message is correct. The issue is with the grok custom pattern feature. For some reason the same pattern is splitting the message into fields but when it gets to the grok filter it does not work.

[Christian_Dahlqvist]

I am not sure I follow. Can you show anevent that has been indexed and not been parsed correctly?

[zozo6015 (Peter)]

Here you have a link to an event that is not splitted into fields.

https://pastebin.com/z1LRBQPx

I think it might be the case of the message not interpretting the \n while the grok considers that as a word and not a new line.

[Badger]

@zozo6015

pattern_definitions => { "GREEDYMULTILINE" => "(.|\r|\n)*", "THREADNAME" => "[^"]+" }

You cannot have a comma between patterns. It will not compile.

If you have a literal newline in your data then use one in the pattern

"%{MONTH:month} %{MONTHDAY:day}, %{YEAR:year} %{TIME:time} %{WORD:day_period} %{NOTSPACE:[system][jetty][class]} %{WORD:[system][jetty][method]}
%{GREEDYMULTILINE:multiline}"

[zozo6015 (Peter)]

Changed the config but still not splitting up the message fields on smaller fields.

 grok {
   match => { "message" => ["%{MONTH:month} %{MONTHDAY:day}, %{YEAR:year} %{TIME:time} %{WORD:day_period} %{NOTSPACE:[system][jetty][class]} %{WORD:[system][jetty][method]}(:?\n|\\n)%{GREEDYMULTILINE:multiline}",
                           "%{WORD:severity} in thread \"%{THREADNAME:threadName}\" %{GREEDYDATA:[system][jetty][data]}",
                           "%{WORD:severity} in thread \"%{THREADNAME:threadName}\" %{NOTSPACE:[system][jetty][exceptionClass]}: %{GREEDYDATA:exceptionMessage}(:?\n|\\n) %{GREEDYMULTILINE:exceptionMultiline}"
                           ] }
   pattern_definitions => {
     "GREEDYMULTILINE" => "(.|\r|\n)*"
     "THREADNAME" => "[^\"]+"
   }
   remove_field => "host"
}