Hi there,
we're using Logstash to read some files and then match these with grok. So far so fine.
Our Filters:
grok {
tag_on_failure => ["tomcat_match_failed"]
match => [ "message", "%{TIMESTAMP_ISO8601:timestamp} %{DATA:thread} %{LOGLEVEL:loglevel} %{DATA:origin}\[(?:RID:%{DATA:request_id})?(?: ?NID:%{DATA:hostname})?(?: ?SID:?%{DATA:sessionid})?(?:\.%{DATA:route})?(?: ?CID:%{DATA:customer_uuid})?\]: (?m)%{GREEDYDATA:log}" ]
}
This works nice, because we don't get any "tomcat_match_failed" tags.
But right after the "grok"-Filter is a "date-Filter:
date {
match => [ "timestamp", "ISO8601" ]
}
I can't use "tag_on_failure" here, because of the missing support of this flag.
Now we get "_grokparsefail"-tags in all of our logs.
With this "date"-Filter we want to make sure the @timestamp field is correctly filled with the event-date.
A typical logevent begins with this:
2015-06-10T13:40:25,919+0200
You might notice the "," as decimal divisor... which should be supported, at least due to https://grokdebug.herokuapp.com/patterns#
YEAR (?>\d\d){1,2}
HOUR (?:2[0123]|[01]?[0-9])
MINUTE (?:[0-5][0-9])
# '60' is a leap second in most time standards and thus is valid.
SECOND (?:(?:[0-5][0-9]|60)(?:[:.,][0-9]+)?)
TIME (?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9])
# datestamp is YYYY/MM/DD-HH:MM:SS.UUUU (or something like it)
DATE_US %{MONTHNUM}[/-]%{MONTHDAY}[/-]%{YEAR}
DATE_EU %{MONTHDAY}[./-]%{MONTHNUM}[./-]%{YEAR}
ISO8601_TIMEZONE (?:Z|[+-]%{HOUR}(?::?%{MINUTE}))
ISO8601_SECOND (?:%{SECOND}|60)
TIMESTAMP_ISO8601 %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}?
DATE %{DATE_US}|%{DATE_EU}
DATESTAMP %{DATE}[- ]%{TIME}
TZ (?:[PMCE][SD]T|UTC)
DATESTAMP_RFC822 %{DAY} %{MONTH} %{MONTHDAY} %{YEAR} %{TIME} %{TZ}
DATESTAMP_OTHER %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{TZ} %{YEAR}
What is the best way to fix this situation?
If the following is right, then the documentation of the date-Filter is too unspecific:
date {
match => [ "timestamp", "TIMESTAMP_ISO8601" ]
}