How do I match a log pattern to different log files?

Elastic Stack Logstash
1

I have two patterns defined for two different kind of logs in my config file.. example: simple_logs and error_logs

So, now I am adding a new field, if the pattern matches simple-logs.
But even if my log file contains the error log, this matches with the simple log pattern and adds the new field.

How can this be resolved as I want to add a new field only to simple logs and show an exception if error logs are not matched with the pattern?

Below snippet shows the filter of my config file:
filter
{
if [type] == "logfile"
{
grok {
patterns_dir => "D:/Logstash/patterns"
match => [ "message", "%{SIMPLE_PATTERN}" ]
add_field => { "log" => "simplelogs" }
}
}
}

This is the error log to which it is putting a new field "log" => "simplelogs".. which was defined for the other simple pattern below it:
{
"message" => "java.lang.Exception: 2012-02-03 19:11:02 SampleClass8 [WA
RN] problem finding id 153454612 at com.osa.mocklogger.MockLogger$2.run(MockLogg
er.java:83)\r",
"@version" => "1",
"@timestamp" => "2015-09-04T08:38:59.922Z",
"host" => "D-113044563",
"path" => "D:/Logstash/log_file/logFile.log",
"type" => "logfile",
"time" => "2012-02-03 19:11:02",
"samplenumber" => "8",
"info" => "WARN",
"all" => "problem finding id 153454612 at com.osa.mocklogger.MockLo
gger$2.run(MockLogger.java:83)\r",
"log" => "simplelogs"
}

{
"message" => "2012-02-03 18:35:34 SampleClass0 [ERROR] incorrect id 18
86438513\r",
"@version" => "1",
"@timestamp" => "2015-09-04T08:38:59.922Z",
"host" => "D-113044563",
"path" => "D:/Logstash/log_file/logFile.log",
"type" => "logfile",
"time" => "2012-02-03 18:35:34",
"samplenumber" => "0",
"info" => "ERROR",
"all" => "incorrect id 1886438513\r",
"log" => "simplelogs"
}

2

Hi!

If your message does not match the pattern, grok will add a tag _grokparsefailure to your event.

Then you can test if this tag exist:

if "_grokparsefailure" in [tags] {
  ...
}

However, with your configuration, it will put the new field log even if your event does not match the pattern.

You can use mutate after testing the message type, for example:

filter {
  if [type] == "logfile" {
    grok	{
      patterns_dir => "D:/Logstash/patterns"
      match => [ "message", "%{SIMPLE_PATTERN}" ]
    }
    if [info] == "INFO" {
      mutate {
        add_field => { "log" => "simplelogs" }
      }
    }
    if "_grokparsefailure" in [tags] {
      ...
    }
  }
}

Hope it will help you :wink: