Stumped on TIMESTAMP_ISO8601 in grok

Elastic Stack Logstash
1

Hello,

I'm not sure why this is throwing an error.

the input file:
<133>2016-08-10T16:33:39.827713+00:00 localhost opt-log-nginx-access

the configfile:
input {
stdin { type => syslog }
}

filter {
  if [type] == "syslog" {
    grok {
      match => [ "message", "%{SYSLOG5424PRI:priority}%{TIMESTAMP_ISO8601:timestamp} %{IPORHOST:myhost}" ]
    }
    date { match => [ "timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss", "MMM dd HH:mm:ss Z", "MMM d HH:mm:ss", "dd/MMM/yyyy:HH:mm:ss Z", "TIMESTAMP_ISO8601" ] }
    mutate {
        add_field => { "imjava" => "imhere" }
        add_tag => "imatag"
    }
  }
}

output {
  stdout { codec => rubydebug }
  file { path => "/tmp/blah.txt" }
}

grok pattern per grok debug
TIMESTAMP_ISO8601 %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}?

grok debug says I'm good, but I get this error msg:

Settings: Default pipeline workers: 1
Pipeline aborted due to error {:exception=>#<LogStash::ConfigurationError: Cannot register filter date plugin. The error reported is:
  Illegal pattern component: T for pattern 'TIMESTAMP_ISO8601'>, :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-date-2.1.6/lib/logstash/filters/date.rb:297:in `setupMatcher'", "org/jruby/RubyArray.java:1613:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-date-2.1.6/lib/logstash/filters/date.rb:224:in `setupMatcher'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-date-2.1.6/lib/logstash/filters/date.rb:189:in `register'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:182:in `start_workers'", "org/jruby/RubyArray.java:1613:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:182:in `start_workers'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:136:in `run'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/agent.rb:473:in `start_pipeline'"], :level=>:error}
stopping pipeline {:id=>"main"}

important part being:
The error reported is: Illegal pattern component: T for pattern 'TIMESTAMP_ISO8601'

any ideas?

thanks!

2

It's not grok that's complaining, it's date. TIMESTAMP_ISO8601 is a grok pattern, not a date pattern. Try ISO8601 instead.

1 Like
3

Many thanks for that reply!

So, in my mind it'd seems only logical that date would also have the same patterns as grok from the standard library. At least those related to time & date. Would it be worth while for me to file an enhancement request on github for this?

thanks again!

4

It's not up to me, but I can see a number of problems with supporting grok patterns in the date filter. Not so much because it's technically hard but because creating a reasonable user interface without too many corner cases and contradictions could be challenging and in the end not really lead to an improvement. The Logstash team might see things differently, so filing an issue might not be in vain.