Is there a way to tag for different grok matches?

Elastic Stack Logstash
1

I am using the GELF Docker driver to send logs to Logstash. It has been working well so far but I am having trouble finding a good way to separate nginx access logs and nginx error logs. I was thinking that I would just add a take for each match like the following:

  if [image_name] =~ /nginx/ {
    grok {
        match => [ "message", "%{INT:status} - %{NOTSPACE:referrer} - %{NOTSPACE:domain} - %{IPORHOST:clientip} - %{QS:request} %{INT:body_bytes_sent} %{QS:http_referer}" ]
        add_tag => [ "nginx_access" ]
    }
    grok {
        match => [ "message", "(?<timestamp>%{YEAR}[./]%{MONTHNUM}[./]%{MONTHDAY} %{TIME}) \[%{LOGLEVEL:severity}\] %{POSINT:pid}#%{NUMBER}: \*%{NUMBER:tid} %{GREEDYDATA:errormessage}, client: %{IP:client}" ]
        add_tag => [ "nginx_err" ]
    }
  }

This matches the first pattern, adds the tag but then fails on the second pattern and I end up with a _grokparsefailure. What's a good way to match and tag access/error logs correctly with GELF?

2

Wrap all but the first grok filter in a conditional so that the subsequent ones are only tried if there hasn't been a match earlier.

grok {
  ...
}
if "_grokparsefailure" in [tags] {
  grok {
    ...
    remove_tag => ["_grokparsefailure"]
  }
}
3

Thank, I'll give this a try. Will this work for a scenario where I have many grok patterns to try and many different tags to apply?

4

Sure, it'll work. Matching lots and lots of regexps against all messages is of course computationally expensive so you'll want to order your expressions wisely.

5

Makes sense. Thanks again.