Grok Parsing for multiple patterns in single log file

[venkatkumar229 (Venkat Kumar)]

Hi Everyone,

I am writing a grok pattern to parse a log file where i have two types of log lines.

1. 2022-04-13 06:38:24.472 DBG Microsoft.AspNet.server --- Connection Id "Xxxxxxxx" accepted

2. 2022-04-13 06:38:19.330 dd513002748agdv-2-244-avsgg PRAVEEN1234:000000003 DBG Microsoft.AspNet.server --- Health check api health check completed

In 2nd log pattern we have extra 2 strings after datestamp where as we don't have those two strings in 1 pattern.

Can some one please help me in creating a combined grok pattern for this log file.

Grok which i have tried but not working properly.

GROK:
%{DATESTAMP:date}%{SPACE}(%{NOTSPACE:id}%{SPACE}%{NOTSPACE:user}|%{SPACE})%{SPACE}%{LOGLEVEL:log_level}%{SPACE}%{NOTSPACE: service}%{SPACE}---%{SPACE}%{GREEDYDATA: response}

O/p:

Regards,
Praveen Kumar

[Badger]

@venkatkumar229

%{DATESTAMP:date}%{SPACE}(%{NOTSPACE:id}%{SPACE}%{NOTSPACE:user}|%{SPACE})%{SPACE}%{LOGLEVEL:log_level}%{SPACE}%{NOTSPACE: service}%{SPACE}---%{SPACE}%{GREEDYDATA: response}

Well, I learned something new today! %{SPACE} matches \s*. That is, zero or more whitespace characters. So "DEBUGINFO" (with no space) will match "%{LOG_LEVEL}%{SPACE}%{LOG_LEVEL}".

That can result in "Microsoft.AspNet.server" matching %{NOTSPACE:id}%{SPACE}%{NOTSPACE:user} and user being set to "er"!

It also explains why %{SPACE}(%{NOTSPACE:id}%{SPACE}%{NOTSPACE:user}|%{SPACE})%{SPACE}%{LOGLEVEL:log_level}, which sometimes reduces to %{SPACE}%{SPACE}%{SPACE}%{LOGLEVEL:log_level} can match when there is only a single space.

We can fix things by changing some of those %{SPACE} to \s+ (one or more whitespace characters, rather than zero or more). Also you have spaces in %{NOTSPACE: service} and %{GREEDYDATA: response} which does something weird that I do not understand. You need to change

%{NOTSPACE: service}

to

%{NOTSPACE:service}

Try

"%{DATESTAMP:date}%{SPACE}(%{NOTSPACE:id}\s+%{NOTSPACE:user}\s+)?%{LOGLEVEL:log_level}%{SPACE}%{NOTSPACE:service}%{SPACE}---%{SPACE}%{GREEDYDATA:response}"

Also, DBG does not match LOGLEVEL, so you may need a

 mutate { gsub => [ "message", "DBG", "DEBUG" ] }

[venkatkumar229 (Venkat Kumar)]

Thanks alot Badger...Your solution worked perfectly.