How to parse log file with different types of messages

Elastic Stack Logstash
1

Hi everyone,

I have a log file which contains complicated message types. Here is an example:

2016-07-07 13:30:02 [UnzipFile] Before file collection
2016-07-07 13:30:02 [GetZipCol] Start get sorted zip file collection
2016-07-07 13:30:02 [GetZipCol] ProcessDate: 2016-07-07
2016-07-07 13:30:02 [GetZipCol] End get sorted zip file collection
2016-07-07 13:30:02 [Main] [ERROR] No unzip file

The following grok pattern is only suitable for first 4 lines but not last line.

grok{
	match => {"message" => ['%{Date:Date}%{SPACE}%{Time:Time}%{SPACE}%{WORD:Job}%{SPACE}%{GREEDYDATA:Message}']}
    	}

I would like to know how should I modify the grok pattern as to capture[ERROR] from the last message. Is there anyone know how the way to do this?

Thanks.

2

I'm surprised that WORD works for matching [UnzipFile]. Assuming you don't really want the square brackets in your fields, this untested example removes them and supports the optional [ERROR]:

%{Date:Date}%{SPACE}%{Time:Time}%{SPACE}\[%{WORD:Job}\](%{SPACE}\[%{WORD:whatever}\])?%{SPACE}%{GREEDYDATA:Message}

The key here is (...)? for making a group of tokens optional.

You can also list multiple grok expressions in the same filter and Logstash will try them all in order and break when it gets a match. There's an example of this in the documentation.