I want to match two grok patterns in one config file?

[Yaswanth]

Hi,

I have two types of error messages in the below format:

[2017-05-25 01:00:00,647][ERROR][marvel.agent.exporter.local] local exporter [default_local] - failed to delete indices
RemoteTransportException[[data-0][10.0.0.8:9300][indices:admin/delete]]; nested: IndexNotFoundException[no such index];
...
[2017-05-18 00:00:06,339][DEBUG][action.admin.indices.create] [data-2] [data-may-2017,data-apr-2017,data-mar-2017] failed to create
[data-may-2017,data-apr-2017,data-mar-2017] InvalidIndexNameException[Invalid index name ..

My logstash configuration is like this:

input {
      file {
            path => "D:\logstash\logstash-2.4.0\bin\logs.txt"
            start_position => "beginning"
	    codec => multiline {
            pattern => "^\[%{TIMESTAMP_ISO8601:TIMESTAMP}\]"
            negate => true
            what => "previous"
        }
  }

}
filter {
   grok {
        match => [ "message", "(?m)^\[%{TIMESTAMP_ISO8601:TIMESTAMP}\]\[%{LOGLEVEL:LEVEL}%{SPACE}\]\[%{DATA:ERRORTYPE}\]%{SPACE}\[%{DATA:SERVERNAME}\]%{SPACE}(?<ERRORMESSAGE>(.|\r|\n)*)", "message",  "(?m)^\[%{TIMESTAMP_ISO8601:TIMESTAMP}\]\[%{LOGLEVEL:LEVEL}%{SPACE}\]\[%{DATA:ERRORTYPE}%{SPACE}\]%{SPACE}(?<ERRORMESSAGE>(.|\r|\n)*)"]
   }
mutate {
    remove_field => ["message","@version","path","host" ]
  }

}
output {
 
  stdout { codec => rubydebug }
}

For Both the logs it is taking only the first grok pattern. Why it is not taking the second one?

Thanks

[Jaxon_Kochel]

Im not entirely sure, but when I do multiple patters i do:

grok{

match => [ "message" ... # first pattern in one match
match = > [ "message" ... #second pattern in another

}

Also I didn't look to closely, but if both entries match the same pattern then they'll always go to that one pattern and not the other.

[Yaswanth]

Thanks @Jaxon_Kochel

I think what you are saying is right . But how can i overcome this?

Whether i have to give like this

Filter{
grok{
match=>"msg"
}
if [LEVEL] == "Error"
grok {
match = > "msg"
}
else {
drop{}
}

Thanks

[Jaxon_Kochel]

Use this pattern for when you have a servername:

(?m)^[%{TIMESTAMP_ISO8601:TIMESTAMP}][ERROR][%{DATA:ERRORTYPE}]%{SPACE}[%{URIHOST:SERVERNAME}]%{SPACE}(?(.|\r|\n)*)

Ichanged it form being DATA to URIHOST

This prevents a non servername log form being matched and will solve the issue

[Yaswanth]

Thanks @Jaxon_Kochel

I used your suggestion by giving grok like this:

filter {
   grok {
        match => [ "message", "(?m)^\[%{TIMESTAMP_ISO8601:TIMESTAMP}\]\[%{LOGLEVEL:LEVEL}%{SPACE}\]\[%{DATA:ERRORTYPE}\]%{SPACE}\[%{URIHOST:SERVERNAME}\]%{SPACE}(?<ERRORMESSAGE>(.|\r|\n)*)", "message",  "(?m)^\[%{TIMESTAMP_ISO8601:TIMESTAMP}\]\[%{LOGLEVEL:LEVEL}%{SPACE}\]\[%{DATA:ERRORTYPE}%{SPACE}\]%{SPACE}(?<ERRORMESSAGE>(.|\r|\n)*)"]
   }
mutate {
    remove_field => ["message","@version","path","host" ]
  }

} 

But still i am getting the same response.

But i used the these two grok patterns in Grok constructor site it is working fine(i.e.First grok only matching second log and second grok only matching first log) .

Thanks

[Yaswanth]

Finally came with a filter that satisfies my requirement.

input {
      file {
            path => "D:\logstash\logstash-2.4.0\bin\logs.txt"
            start_position => "beginning"
             type => "log"
        codec => multiline {
            pattern => "^\[%{TIMESTAMP_ISO8601:TIMESTAMP}\]"
            negate => true
            what => "previous"
        }
  }

}
    filter {
      if [type] == "log" {
        grok {
          match => [ "message", "(?m)^\[%{TIMESTAMP_ISO8601:TIMESTAMP}\]\[%{LOGLEVEL:LEVEL}%{SPACE}\]\[%{DATA:ERRORTYPE}%{SPACE}\]%{SPACE}(?<ERRORMESSAGE>(.|\r|\n)*)"]
        }
    # DEBUG Logs
    if "grokked" not in [tags] and "DEBUG" == [LEVEL] {
    grok { match => [ "ERRORMESSAGE", "(?m)^\[%{DATA:SERVERNAME}\]" ]
    add_tag => [ "Debug Logs", "grokked" ]
    tag_on_failure => [ ]
    }
    }
    }
    }
output {

  stdout { codec => rubydebug }
}

Thanks