Logstash - Multiple grok pattern not working together

Elastic Stack Logstash
1

I am very new in using Logstash. I have two kinds of log,

Pattern 1 : --2019-05-09 08:53:45.057 -INFO 11736 --- [ntainer#1-0-C-1] c.s.s.service.MessageLogServiceImpl : [adc7fd862db5307a688817198046b284dbb12b9347bed9067320caa49d8efa381557392024151] Event => Message Status Change [Start Time : 09052019 08:53:44] : CUSTOM_PROCESSING_COMPLETED

Pattern 2 : --2019-05-09 06:49:05.590 -TRACE 6293 --- [ntainer#0-0-C-1] c.s.s.service.MessageLogServiceImpl : [41a6811cbc1c66eda0e942712a12a003d6bf4654b3edb6d24bf159b592afc64f1557384545548] Event => Message Failure Identified : INVALID_STRUCTURE

Though there are many more other lines, but I want to consider only these two types. Hence I used below filter,

 grok {
     #Event : message status change
     match => {
         "message" => "--(?<logtime>[^\]]*) -%{LOGLEVEL:level} (?<pid>\d+) --- \[(?<thread>[^\]]+)] (?<classname>[\w.]+)\s+: \[(?<token>[^\]]+)] Event \=> Message Status Change \[Start Time : (?<start>[^\]]*)\] : (?<status>[\w]+)"
     }
     add_field => {
         "event" => "message_status_change"
     }
 }


 grok {
     #Event : message failure
     match => {
         "message" => "--(?<logtime>[^\]]*) -%{LOGLEVEL:level} (?<pid>\d+) --- \[(?<thread>[^\]]+)] (?<classname>[\w.]+)\s+: \[(?<token>[^\]]+)] Event \=> Message Failure Identified : (?<code>[\w]+)"
     }
     add_field => {
         "event" => "message_failure"
     }
 }

I have also noticed that both of these grok patterns work individually (if I comment one, then other one works perfectly). Logstash server also ok when both patterns are active. But it raises a grokparse error when both of them is open and a new line is added in the log file.

Also I want to know, though I am configured the input to read from a file from beginning, it is not reading even after server restart unless I add a new line in the log. Why this behaviour?

Thanks in advance.

2

That is to be expected. If a line matches one of those two grok filters it will not match the other one, so it will get tagged with _grokparsefailure.

If you have a persistent sincedb then start_position is ignored once a sincedb entry exists. It is only used the first time a file is seen. You can avoid persisting the sincedb across restarts using 'sincedb_path => "NUL"' on Windows, or 'sincedb_path => "/dev/null" on UNIX.

3

Thanks for reply. I understand what you said. But whats the solution here? What if I want two parse two different lines? Cant I use two groks together?

4

Yes, you can use multiple groks. If you do not care that a particular grok failed you could add a mutate+remove_tag filter to remove the _grokparsefailure.

Alternatively, you could test the message contents before invoking the grok

if [message] =~ "Message Failure Identified" {
     grok { [...]

Alternatively, the match option can match a line against an array of patterns.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.