_grokparsefailure in the file

[varunverma]

I have the below grok filter in my logstash conf file

filter {
grok {
match => { 'message' => ' %{IPORHOST:clientip} - %{USERNAME:user} %{HTTPDATE:timestamp} \"%{WORD:method} %{NOTSPACE:request} %{NOTSPACE:httpversion} %{NUMBER:httpCode} %{NUMBER:offset} \"%{URI:url}\" \"%{GREEDYDATA:msgLeft} ' }
}
}

the pattern is
0:0:0:0:0:0:0:1 - admin 31/Jul/2017:13:04:17 +1000 "GET /mnt/overlay/granite/ui/content/shell/header/actions/pulse.data.json?_=1501459094213 HTTP/1.1" 200 1337 "http://localhost:4502/sites.html/content/we-retail/ca/en" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"

but I am getting _grokparsefailure which I am not getting in http://grokconstructor.appspot.com/

[Hisushi (Maren Sonnenschein)]

@varunverma

%{IPORHOST:clientip} - %{USERNAME:user} %{HTTPDATE:timestamp} \"%{WORD:method} %{NOTSPACE:request} %{NOTSPACE:httpversion} %{NUMBER:httpCode} %{NUMBER:offset} \"%{URI:url}" \"%{GREEDYDATA:msgLeft}

Hi @varunverma,

I think that the quotation marks in your message might be the problem as you are escaping " instead of ” and “. When looking at their hex value, e.g., here you'll see the difference. As a workaround either replacing them beforehand or escaping these characters using their hex value (\x93) should work.

Best regards,

Hisushi

[varunverma]

Hi

I am using the below filter now but still getting the same grokparsefailure

filter {
grok {
match => { 'message' => '%{IPORHOST:clientip} - %{USERNAME:user} %{HTTPDATE:timestamp} \%{NOTSPACE:method} %{NOTSPACE:request} %{NOTSPACE:httpversion} %{NUMBER:httpCode} %{NUMBER:offset} \%{NOTSPACE:url} \%{GREEDYDATA:data}'}
}

}

Thanks
Varun

[pjanzen (Paul Janzen)]

@varunverma

%{URI:url}

I tested your input string and this works for me.

%{IPORHOST:clientip} - %{USERNAME:user} %{HTTPDATE:timestamp} %{NOTSPACE}%{WORD:method} %{URIPATHPARAM:request} %{NOTSPACE:httpversion} %{NUMBER:httpCode} %{NUMBER:bytes} %{NOTSPACE}%{URI:url}%{NOTSPACE} %{GREEDYDATA:agent}

I tested it on http://grokdebug.herokuapp.com/

If I copy & paste your grok filter there with that data you provide I get a compile error. If I remove the \ from you patern it works there as well..

[Hisushi (Maren Sonnenschein)]

Hi @varunverma,

this won't work as you're escaping % now. Please consider my suggested solution and concentrate on the provided quotation marks in your message.

To make things clear: Either use something like

mutate {
  gsub => { 
      "message", "[\“\”]", "\""
  }
}

to replace your quotation marks (“ - unicode 201C, ” - unicode 201D) or escape these special characters.

Using

%{IPORHOST:clientip} - %{USERNAME:user} %{HTTPDATE:timestamp} \"%{WORD:method} %{NOTSPACE:request} HTTP/%{NOTSPACE:httpversion}\" %{NUMBER:httpCode} %{NUMBER:offset} \"%{URI:url}\" \"%{GREEDYDATA:msgLeft}\"

with

0:0:0:0:0:0:0:1 - admin 31/Jul/2017:13:04:17 +1000 "GET /mnt/overlay/granite/ui/content/shell/header/actions/pulse.data.json?_=1501459094213 HTTP/1.1" 200 1337 "http://localhost:4502/sites.html/content/we-retail/ca/en" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"

works perfectly but doesn't match your original message (note the quotation marks!):

0:0:0:0:0:0:0:1 - admin 31/Jul/2017:13:04:17 +1000 “GET /mnt/overlay/granite/ui/content/shell/header/actions/pulse.data.json?_=1501459094213 HTTP/1.1” 200 1337 “http://localhost:4502/sites.html/content/we-retail/ca/en” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36”

Best regards,

Hisushi

[varunverma]

Thanks Hisushi