Force Line to use perticular grok

Hi experts.
I have my multineline filter as below in my logstash configuration

my sample log lines

  if [doc_type] =~ "Indexer" {
    grok {
    patterns_dir => ["C:/ELK6.2/logstash-6.2.2/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns"]
    match => [ "message", "(?m)%{DatenTime1:DatenTime}%{Unt23:unt1}-\s*%{TypeA:Function_Type}\s*%{TImeS:TImeSpent}%{MorS:Min_or_Sec}\s*%{Objects:Num_Objects}" ]
    match => [ "message", "(?m)%{DatenTime1:DatenTime}%{Unt23:unt1}-\s*%{TypeB:Function_Type}\s*%{TImeS:TImeSpent}%{MorS:Min_or_Sec}\s*%{Objects:Num_Objects}" ]

to filter

2018-01-31 23:05:16,476 INFO  - RevRule        0.099s 0 objects
2018-01-31 23:05:16,501 INFO  - AM impacted 0.025s 0 objects

Where only difference in my grok filter is


Now as per Discussion here the first match wins and giving me a wrong filter value.
Wrong one, how i am getting it

	Function_Type      TImeSpent      Min_or_Sec      	Num_Objects 
    AM impacted         0.025            s  	             0
      RevRule  0         .099            s                   0

how i wish to see

	Function_Type      TImeSpent      Min_or_Sec      	Num_Objects 
    AM impacted         0.025            s  	             0
      RevRule           0.099            s                   0

Any help, how can i overcome this ?


Please post your custom patterns too. Without the patterns it's difficult to debug what happens.

Remark: I was confused about "multiline". I thought you had multiline messages that you are having troubles with. Read about multiline plugin here.

Btw. you could just use an OR operator in the regex for the TypeA|TypeB Function_type field, saving you one line and improving performance, since you'll only run one regex matching instead of two (in case of a TypeB line).

Hi atira,

Thanks for your helping hand, i tried your suggested method, TypeA|TypeB , Didn't got any help.
the problem i am facing is with multiline is the first line is getting WRONGLY resolved with TypeA itself and reporting rest all as wrong, reason why i wish, if there is a way to force each line to specific filter ?

Any Idea ?


Please post your custom patterns.

Also, multi grok patterns should be defined in one array in a [ "field", "pattern", "field", "pattern", "field" ... ] syntax.

if [doc_type] =~ "Indexer" {
    grok {
        patterns_dir => ["C:/ELK6.2/logstash-6.2.2/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns"]
        match => [
           "message", "(?m)%{DatenTime1:DatenTime}%{Unt23:unt1}-\s*%{TypeA:Function_Type}\s*%{TImeS:TImeSpent}%{MorS:Min_or_Sec}\s*%{Objects:Num_Objects}",
           "message", "(?m)%{DatenTime1:DatenTime}%{Unt23:unt1}-\s*%{TypeB:Function_Type}\s*%{TImeS:TImeSpent}%{MorS:Min_or_Sec}\s*%{Objects:Num_Objects}"

Hi atira,
i have tried your another suggestion as well, NO LUCK :frowning:
here is my custom pattern.

DatenTime1 \d{4}-\d{2}-\d{2}\s*([0-1]?\d|2[0-3])(?::([0-5]?\d))?(?::([0-5]?\d)),([0-9]{1,3}\d)
Unt23 (\s*\w*\s*-\s*)
TypeA (\w*)
TypeB (\w*\s*\w*)
TImeS (\d*.\d*)
MorS (\w*)
Objects (\d*)

do you know any other way to force ?

Your custom patterns seem to be too ambiguous, they overlap each other.
\w* - the * means 0 or more, so it will match even if there is 0.
Same for \s*, it will match even if there no space.

If you expect at least one of the given character, use +.
\w+ - means you expect at least one word character
\s+ - means you expect at least one space

However, in your case I would skip all these. Your log entries don't need custom patterns. Here's the whole thing, it will match both lines of your example:

if [doc_type] =~ "Indexer" {
    grok {
        match => [ "message", "%{TIMESTAMP_ISO8601:DatenTime}\s+%{LOGLEVEL:unt1}\s+-\s+%{DATA:Function_Type}\s+%{NUMBER:TimeSpent}(?<Min_or_Sec>[a-zA-Z])\s+%{INT:Num_Objects}\s+objects" ]

You have really solved it. and your previous solution also worked with multiline.

Thank you so much.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.