Getting grokfailure, but grok constructor test works

Elastic Stack Logstash
1

Hello,

I am getting this discrepancy between http://grokconstructor.appspot.com/do/match and logstash grok filter. In the grok constructor site I am getting following parsed:

"10.1.40.20 22.77.167.65 - [29/Jun/2016:19:07:42 +0000] "GET /de-soto-mo/real-estate-appraisers HTTP/1.1" 200 29960 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" "-" 0.701"

using following pattern:

"%{IPORHOST:remote_addr} %{IPORHOST:http_x_forwarded_for} - [%{HTTPDATE:timestamp}]\ \"%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:response} (?:%{NUMBER:bytes}|undefined) \"(%{URI:referrer}|-)\" \"%{DATA:agent}\" \"%{DATA:cookie}\" %{NUMBER:response_time}"

When I run logstash with following grok filter

       grok {
            break_on_match => false
            match => [ "message", "%{IPORHOST:remote_addr} %{IPORHOST:http_x_forwarded_for} - \[%{HTTPDATE:timestamp}]\ \\"%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\\" %{NUMBER:response} (?:%{NUMBER:bytes}|undefined) \\"(%{URI:referrer}|-)\\" \\"%{DATA:agent}\\" \\"%{DATA:cookie}\\" %{NUMBER:response_time}" ]
           }

I am getting

{
"@timestamp" => "2016-06-29T19:07:43.060Z",
"input_type" => "log",
"message" => "10.1.40.20 22.77.167.65 - [29/Jun/2016:19:07:42 +0000] "GET /de-soto-mo/real-estate-appraisers HTTP/1.1" 200 29960 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" "-" 0.701",
"offset" => 8039375,
"source" => "/usr/local/nextgen/ypu-logs/access.log",
"type" => "nginx",
"@version" => "1",
"kafka" => {
"msg_size" => 448,
"topic" => "ypu-logs-wc1",
"consumer_group" => "logstash",
"partition" => 7,
"key" => nil
},
"nginx_host" => "blah.com",
"tags" => [
** [0] "_grokparsefailure"**
]
}

Any reason why this is the case?

2

Start with the simplest possible expression, %{IPORHOST:remote_addr} and see if it works. If yes, add the next token to the end of the expression. If no, you've found the problem. Repeat.

3

That's what I have done on the grok testing site, after which I have implemented this in logstash. And that's where it's failing. I don't have any custom patterns