Matching with "date" creates _grokparsefailure but matches are okay

Elastic Stack Logstash
1

Hi there,

we're using Logstash to read some files and then match these with grok. So far so fine.

Our Filters:

grok {
    tag_on_failure => ["tomcat_match_failed"]
    match => [ "message", "%{TIMESTAMP_ISO8601:timestamp} %{DATA:thread} %{LOGLEVEL:loglevel} %{DATA:origin}\[(?:RID:%{DATA:request_id})?(?: ?NID:%{DATA:hostname})?(?: ?SID:?%{DATA:sessionid})?(?:\.%{DATA:route})?(?: ?CID:%{DATA:customer_uuid})?\]: (?m)%{GREEDYDATA:log}" ]
    }

This works nice, because we don't get any "tomcat_match_failed" tags.
But right after the "grok"-Filter is a "date-Filter:

date {
  match => [ "timestamp", "ISO8601" ]
  }

I can't use "tag_on_failure" here, because of the missing support of this flag.
Now we get "_grokparsefail"-tags in all of our logs.

With this "date"-Filter we want to make sure the @timestamp field is correctly filled with the event-date.

A typical logevent begins with this:

2015-06-10T13:40:25,919+0200

You might notice the "," as decimal divisor... which should be supported, at least due to https://grokdebug.herokuapp.com/patterns#

YEAR (?>\d\d){1,2}
HOUR (?:2[0123]|[01]?[0-9])
MINUTE (?:[0-5][0-9])
# '60' is a leap second in most time standards and thus is valid.
SECOND (?:(?:[0-5][0-9]|60)(?:[:.,][0-9]+)?)
TIME (?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9])
# datestamp is YYYY/MM/DD-HH:MM:SS.UUUU (or something like it)
DATE_US %{MONTHNUM}[/-]%{MONTHDAY}[/-]%{YEAR}
DATE_EU %{MONTHDAY}[./-]%{MONTHNUM}[./-]%{YEAR}
ISO8601_TIMEZONE (?:Z|[+-]%{HOUR}(?::?%{MINUTE}))
ISO8601_SECOND (?:%{SECOND}|60)
TIMESTAMP_ISO8601 %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}?
DATE %{DATE_US}|%{DATE_EU}
DATESTAMP %{DATE}[- ]%{TIME}
TZ (?:[PMCE][SD]T|UTC)
DATESTAMP_RFC822 %{DAY} %{MONTH} %{MONTHDAY} %{YEAR} %{TIME} %{TZ}
DATESTAMP_OTHER %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{TZ} %{YEAR}

What is the best way to fix this situation?

If the following is right, then the documentation of the date-Filter is too unspecific:

date {
    match => [ "timestamp", "TIMESTAMP_ISO8601" ]
  }
2

The date filter isn't grok-based. The "ISO8601" tag is a special case that's translated into Joda-Time patterns like this:

Note that commas aren't allowed. I think there's a bug and/or pull request to fix this.

3

Hi Magnus,
ok, then the docs on "date" are right...

I can't see a pull request, but will create one.

Thank you!

Edit:

  • Pull Request created

Using a workaround:

date {
    match => [ "timestamp", "YYYY-MM-dd HH:mm:ssZZ", "YYYY-MM-dd HH:mm:ssZ", "YYYY-MM-dd HH:mm:ss", "YYYY-MM-dd HH:mm:ss,SSSZZ", "YYYY-MM-dd HH:mm:ss,SSSZ", "YYYY-MM-dd HH:mm:ss,SSS", "YYYY-MM-dd HH:mm:ss:SSSZZ", "YYYY-MM-dd HH:mm:ss:SSSZ", "YYYY-MM-dd HH:mm:ss:SSS", "ISO8601" ]
  }