How to resolve `_grokparsefailure` in node.js logs

Elastic Stack Logstash
1

I am new to ELK and trying to instrument a simple node.js app that is using Winston for logging. I have tried a variety of grok patterns (including just %COMBINEDAPACHELOG) but can not seem to get all the data that I need. I have read through the related parse failure posts but am not making traction. Any ideas on how to consistently parse these logs?

Sample log

{"level":"info","message":"127.0.0.1 - - [Tue, 13 Oct 2015 05:19:33 GMT] \"GET /blah HTTP/1.1\" 500 1402 \"-\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:41.0) Gecko/20100101 Firefox/41.0\"\n","timestamp":"2015-10-13T05:19:33.571Z"}

Config File Filter

filter {
  grok {
    match => { "message" => "%{WORD:loglevel}%{WORD:logtype} %{IP:client}%{SPACE}-%{SPACE}-%{SPACE} %{TIME:timestamp} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:response}-%{QUOTEDSTRING},%{TIME:timestamp2}" }
  }
}

Output When Logstash Starts

{
         "level" => "info",
       "message" => "127.0.0.1 - - [Tue, 13 Oct 2015 05:19:33 GMT] \"GET /blah HTTP/1.1\" 500 1402 \"-\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:41.0) Gecko/20100101 Firefox/41.0\"\n",
     "timestamp" => "2015-10-13T05:19:33.571Z",
      "@version" => "1",
    "@timestamp" => "2015-10-13T21:20:38.829Z",
          "host" => "<user>.local",
          "path" => "/Users/<user>/Documents/node-postgres-todo/logs/test-log.log",
          "tags" => [
        [0] "_grokparsefailure"
    ]
}
2

http://grokdebug.herokuapp.com is always super useful for debugging patterns like this!

3

The grok expression above does not match the input. It ends with %{NUMBER:response}-%{QUOTEDSTRING},%{TIME:timestamp2} which bears no resemblance with the actual string.

The message field's contents is deceptively close to COMBINEDAPACHELOG but the date format doesn't match HTTPDATE. You'll have to copy the definition of COMBINEDAPACHELOG (and transitively COMMONAPACHELOG) but replace HTTPDATE with something that matches this date format. DATESTAMP_RFC822 is very close but the timezone name (GMT) won't match the TZ pattern.

4

Thanks guys. I am going to try a different logger as that is what I was afraid of. @warkolm does Logstash have any plans to offer an official grok tool? I tried grokdebug and it wasn't very helpful to just get up and going.