Force Line to use perticular grok

Elastic Stack Logstash
1

Hi experts.
I have my multineline filter as below in my logstash configuration

my sample log lines

  if [doc_type] =~ "Indexer" {
    grok {
    patterns_dir => ["C:/ELK6.2/logstash-6.2.2/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns"]
    match => [ "message", "(?m)%{DatenTime1:DatenTime}%{Unt23:unt1}-\s*%{TypeA:Function_Type}\s*%{TImeS:TImeSpent}%{MorS:Min_or_Sec}\s*%{Objects:Num_Objects}" ]
    match => [ "message", "(?m)%{DatenTime1:DatenTime}%{Unt23:unt1}-\s*%{TypeB:Function_Type}\s*%{TImeS:TImeSpent}%{MorS:Min_or_Sec}\s*%{Objects:Num_Objects}" ]

to filter

2018-01-31 23:05:16,476 INFO  - RevRule        0.099s 0 objects
2018-01-31 23:05:16,501 INFO  - AM impacted 0.025s 0 objects

Where only difference in my grok filter is

TypeA:Function_Type
TypeB:Function_Type

Now as per Discussion here the first match wins and giving me a wrong filter value.
Wrong one, how i am getting it

	Function_Type      TImeSpent      Min_or_Sec      	Num_Objects 
    AM impacted         0.025            s  	             0
      RevRule  0         .099            s                   0

how i wish to see

	Function_Type      TImeSpent      Min_or_Sec      	Num_Objects 
    AM impacted         0.025            s  	             0
      RevRule           0.099            s                   0

Any help, how can i overcome this ?

BR

2

Please post your custom patterns too. Without the patterns it's difficult to debug what happens.

Remark: I was confused about "multiline". I thought you had multiline messages that you are having troubles with. Read about multiline plugin here.

Btw. you could just use an OR operator in the regex for the TypeA|TypeB Function_type field, saving you one line and improving performance, since you'll only run one regex matching instead of two (in case of a TypeB line).

3

Hi atira,

Thanks for your helping hand, i tried your suggested method, TypeA|TypeB , Didn't got any help.
the problem i am facing is with multiline is the first line is getting WRONGLY resolved with TypeA itself and reporting rest all as wrong, reason why i wish, if there is a way to force each line to specific filter ?

Any Idea ?

BR

4

Please post your custom patterns.

Also, multi grok patterns should be defined in one array in a [ "field", "pattern", "field", "pattern", "field" ... ] syntax.

if [doc_type] =~ "Indexer" {
    grok {
        patterns_dir => ["C:/ELK6.2/logstash-6.2.2/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns"]
        match => [
           "message", "(?m)%{DatenTime1:DatenTime}%{Unt23:unt1}-\s*%{TypeA:Function_Type}\s*%{TImeS:TImeSpent}%{MorS:Min_or_Sec}\s*%{Objects:Num_Objects}",
           "message", "(?m)%{DatenTime1:DatenTime}%{Unt23:unt1}-\s*%{TypeB:Function_Type}\s*%{TImeS:TImeSpent}%{MorS:Min_or_Sec}\s*%{Objects:Num_Objects}"
           ]
    }
}
5

Hi atira,
i have tried your another suggestion as well, NO LUCK :frowning:
here is my custom pattern.

DatenTime1 \d{4}-\d{2}-\d{2}\s*([0-1]?\d|2[0-3])(?::([0-5]?\d))?(?::([0-5]?\d)),([0-9]{1,3}\d)
Unt23 (\s*\w*\s*-\s*)
TypeA (\w*)
TypeB (\w*\s*\w*)
TImeS (\d*.\d*)
MorS (\w*)
Objects (\d*)

do you know any other way to force ?

6

Your custom patterns seem to be too ambiguous, they overlap each other.
\w* - the * means 0 or more, so it will match even if there is 0.
Same for \s*, it will match even if there no space.

If you expect at least one of the given character, use +.
\w+ - means you expect at least one word character
\s+ - means you expect at least one space

However, in your case I would skip all these. Your log entries don't need custom patterns. Here's the whole thing, it will match both lines of your example:

if [doc_type] =~ "Indexer" {
    grok {
        match => [ "message", "%{TIMESTAMP_ISO8601:DatenTime}\s+%{LOGLEVEL:unt1}\s+-\s+%{DATA:Function_Type}\s+%{NUMBER:TimeSpent}(?<Min_or_Sec>[a-zA-Z])\s+%{INT:Num_Objects}\s+objects" ]
    }
}
7

You have really solved it. and your previous solution also worked with multiline.

Thank you so much.