How to solve grokparsefailure in Logstash

Elastic Stack Logstash
1

Hi,

I am trying to parse nginx error logs in Logstash. I have tested the grok pattern on http://grokdebug.herokuapp.com/ and it does not give any error. But when I try to use the same log sample and pattern in logstash, I am getting grokparsefailure. What could be the reason for this? Can someone help me with this error?

sample log:
2016/06/23 12:14:41 [warn] 4444#0: *66666 an upstream response is buffered to a temporary file /var/cache/nginx/proxy_temp/2/93/0000004932 while reading upstream , client: 111.111.11.11 , server: xyzr.com, request: "GET /assets/571e6fd4dd7a1e511ca4923c HTTP/1.1", upstream: "http://127.0.0.1:3000/assets/571e6fd4dd7a1e511ca4923c", host: "xyzr.com", referrer: "http://xxx.com/"

logstash conf file:

   file {
        type => "nginx"
 path => "/hands-on-workshop.tar/hands-on-workshop/sample/nginx_error.log"
        start_position => "beginning"
        sincedb_path => "/dev/null"
  }
}

filter {
   if [type] == "nginx" {
    grok {
        match => { "message" => "(?<timestamp>%{YEAR}[./-]%{MONTHNUM}[./-]%{MONTHDAY}[- ]%{TIME}) \[%{LOGLEVEL:severity}\] %{POSINT:pid}#%{NUMBER}: %{GREEDYDATA:errormessage} (?:, client: (?<client>%{IP})) (?:, server: %{IPORHOST:server})(?:, request: %{QS:request})?(?:, upstream: \"%{URI:upstream}\")?(?:, host: %{QS:host})?(?:, referrer: \"%{URI:referrer}\")" }
        remove_tag => ["_grokparsefailure"]
        add_tag => ["nginx_error"]
    }
    date {
    match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ]
    locale => en
  }

  geoip {
    source => "clientip"
  }

  useragent {
        source => "agent"
        target => "useragent"
        }
  }
}


output {
  stdout { codec => rubydebug }

}

thanks.

2

Aren't you missing a ? at the very end of the expression, to make the referred matching optional?

Simplify your expression until it starts working again. Start by removing all the key/value pairs at the end so that the expression ends with GREEDYDATA. Regardless of whether that works or not you know which half of the expression that's problematic.