Logstash Grok pattern match issue


I am facing an issue with parsing data using Logstash using Grok filter, below is the use-case

There are two types of logs I am trying to parse,
//Grok patterns
04-16 07:16:35,359[ INFO]:Except:Processing Application - End for AAAA11112021-04-16-
04-16 12:29:30,044[ INFO]:Except:Processing Application - End for BBBB2222 002021-04-16-

If you notice both logs are identical, the only difference is for one the applicationId has a space and the other does not have a space

I have below two patterns defined for both types of logs

Grok pattern Type1 logs:
%{GREEDYDATA:logdate}[ %{LOGLEVEL:loglevel}]:%{WORD:class}:%{WORD:log_text_1} %{WORD:log_text_2} - %{WORD:log_text_3} %{WORD:log_text_4} %{NOTSPACE:application_id}

Grok pattern Type2 logs:
%{GREEDYDATA:logdate}[ %{LOGLEVEL:loglevel}]:%{WORD:class}:%{WORD:log_text_1} %{WORD:log_text_2} - %{WORD:log_text_3} %{WORD:log_text_4} %{NOTSPACE:application_id} %{NOTSPACE:timestamp}​

The issue is that even for Type2 logs the Type1 pattern matches and I do not get the timestamp, how can I create Grok patterns that differentiate between these two logs?

Any help would be greatly appreciated. Thanks in advance

If you use a grok filter to match against an array of patterns then they are processed in order. Just put the pattern with the space first.

Thank you that worked. How can we remove the grokparsefailures from our logs if we do not want them?

If you want to delete events that have _grokparsefailure then use

if "_grokparsefailure" in [tags] { drop {} }

If you do not want the tag added then use

 tag_on_failure => []

on your grok filter.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.