Grok date parsing trouble

Elastic Stack Logstash
1

Hi Everyone,

I'm trying to parse a tab separated access log and having some trouble with updating the @timestamp field in order to store the actual event time, using logstash 2.2.

Here is a log sample:

2016-03-14 23:05:04 10.0.0.1:8001 GET /transact/images/HealthCheck.gif 10.10.10.1 - 200 0.0010 43 10.10.10.2 -

Here is my filter:

filter {
  if [type] == "wl-access" {
    grok {
      match => { "message" => "%{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day}\t%{TIME:time}\t%{IP:server_ip}:%{NUMBER:server_port}\t%{WORD:method}\t%{URIPATH:uri}\t%{IP:client_ip}\t(?:%{NOTSPACE:uri_query}|-)\t%{NUMBER:status}\t%{NUMBER:response_time}\t%{NUMBER:bytes}\t(?:%{NOTSPACE:user}|)\t(?:%{NOTSPACE:post_args}|)\t%{NOTSPACE:url}\t(?:%{GREEDYDATA:user_agent}|-)" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    mutate {
      add_field => {
        "timestamp" => "%{year}-%{month}-%{day} %{time}"
      }
    }
    date {
      match => [ "timestamp" , "%{TIMESTAMP_ISO8601}" ]
    }
  }
}

This results in the following error:

Error: Cannot register filter date plugin. The error reported is:
Illegal pattern component: T for pattern '%{TIMESTAMP_ISO8601}'

I am however able to successfully parse the value stored in the timestamp field with the grokdebugger, using the %{TIMESTAMP_ISO8601} pattern. Also, when I comment out the date / match part, logstash records the timestamp correctly:

   "timestamp" => "2016-03-14 23:05:04"

I've been at this for several hours, so any feedback would be greatly appreciated :).

Another odd thing is when attempting to add a custom pattern for the long match string, it fails with a grok parsing error. This too parses fine in the grok debugger.. I narrowed it down to the hyphens in '%{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day}', but gave up once I figured out that it works when using it in the config directly. This is less of a problem now, but would be interested to see if anyone experienced this issue.

Thanks so much!

2

I could be wrong but looking at Date filter plugin | Logstash Reference [8.11] | Elastic I think it ought to be:

3

Thanks so much Craig for responding. That allowed logstash to start, but gave the below error when parsing the log.

Failed parsing date from field {:field=>"timestamp", :value=>"2016-03-10 17:00:00", :exception=>"Invalid format: \"2016-03-10 17:00:00\" is too short", :config_parsers=>"ISO8601", :config_locale=>"default=en_US", :level=>:warn}

However, I think I have it working now with reverting back to match => [ "timestamp" , "yyyy-MM-dd HH:mm:ss" ]. I have tried that previously, but before creating the custom timestamp field with mutate, so it wasn't working then.

Here is my filter in case it helps someone, I have added the timezone to the date filter otherwise the time was skewed, and removed the extra fields after the updating @timestamp field. Not the most elegant way I'm sure, but will have to do for now.

`filter {
  if [type] == "wl-access" {
    grok {
      match => { "message" => "%{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day}\t%{TIME:time}\t%{IP:server_ip}:%{NUMBER:server_port}\t%{WORD:method}\t%{URIPATH:uri}\t%{IP:client_ip}\t(?:%{NOTSPACE:uri_query}|-)\t%{NUMBER:status}\t%{NUMBER:response_time}\t%{NUMBER:bytes}\t(?:%{NOTSPACE:user}|)\t(?:%{NOTSPACE:post_args}|)\t%{NOTSPACE:url}\t(?:%{GREEDYDATA:user_agent}|-)" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    mutate {
      add_field => {
        "timestamp" => "%{year}-%{month}-%{day} %{time}"
      }
    }
    date {
      match => [ "timestamp" , "yyyy-MM-dd HH:mm:ss" ]
      timezone => "UTC"
    }
    mutate {
      remove_field => [ "timestamp", "year", "month", "day", "time" ]
    }
  }
}

`

Here is a log sample and output once parsed.

`2016-03-10      17:00:00        10.0.0.53:8001      GET     /transact/images/HealthCheck.gif        10.0.0.142  -       200     0.04743                       10.0.0.139  -`

`{
      "message" => "2016-03-10\t17:00:00\t10.0.0.53:8001\tGET\t/transact/images/HealthCheck.gif\t10.0.0.142\t-\t200\t0.047\t43\t\t\t10.0.0.139\t-\t",
     "@version" => "1",
   "@timestamp" => "2016-03-10T17:00:00.000Z",
         "type" => "wl-access",
         "host" => "loghost",
    "server_ip" => "10.0.0.53",
  "server_port" => "8001",
       "method" => "GET",
          "uri" => "/transact/images/HealthCheck.gif",
    "client_ip" => "10.0.0.142",
    "uri_query" => "-",
       "status" => "200",
"response_time" => "0.047",
        "bytes" => "43",
          "url" => "10.0.0.139",
   "user_agent" => "-\t",
  "received_at" => "2016-03-15T19:55:54.929Z",
"received_from" => "loghost"

}
`