BUG? - ISO8601 Timezone Parsing


I'm trying to parse the /var/log/install.log file on MacOS (I've checked 10.13.x and 10.14.0, they're the same syntax). Unfortunately it's lacking some consistency in date formats between various system logs, but that's not something I can control.

I've got filebeat forwarding to logstash just fine, I'm just having some difficulty with getting the syslog date -> the timestamp field.


2018-09-26 15:12:25+08 iMac.local Installer[99213]: Will use PK session
2018-09-26 15:12:25+08 iMac.local Installer[99213]: Using authorization level of root for IFPKInstallElement

I'm using this grok filter, which fails on the timestamp because as far as I can tell, it's expecting +0800 rather than +08

grok {
match => { "message" => '%{TIMESTAMP_ISO8601:time} %{HOSTNAME:hostname} %{WORD:syslog_program}(?:\[%{NUMBER:syslog_pid}\])?: %{GREEDYDATA:syslog_message}' }

If I change the input text from +08 to +0800 it parses it correctly, which is what I'd expect from the github repo with grok patterns

ISO8601_TIMEZONE (?:Z|[+-]%{HOUR}(?::?%{MINUTE}))

But if I check the ISO8601 standards, it looks like +xx is permitted:

So if the time being described is one hour ahead of UTC (such as the time in Berlin during the winter), the zone designator would be "+01:00", "+0100", or simply "+01".

I've worked around this by overwriting the ISO8601_TIMEZONE pattern in a custom pattern for these log files:

ISO8601_TIMEZONE (?:Z|[+-]%{HOUR}(?::?%{MINUTE})?)

The only change I've made between my pattern declaration and the inbuilt one is to make the non-capture group for minutes optional. Is this a bug? Should I be logging this as an issue in github?


EDIT: Also, the date plugin seems to assume timezone will always be 4 digits, I'm not full bottle enough on logstash yet to know if the date plugin uses the same patterns or not (I assume not), but there may be a need (if this a bug that elastic would want fixed) additional code would need adjustment.

(system) #2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.