I'm trying to parse the /var/log/install.log file on MacOS (I've checked 10.13.x and 10.14.0, they're the same syntax). Unfortunately it's lacking some consistency in date formats between various system logs, but that's not something I can control.
I've got filebeat forwarding to logstash just fine, I'm just having some difficulty with getting the syslog date -> the timestamp field.
Snippits:
2018-09-26 15:12:25+08 iMac.local Installer[99213]: Will use PK session
2018-09-26 15:12:25+08 iMac.local Installer[99213]: Using authorization level of root for IFPKInstallElement
I'm using this grok filter, which fails on the timestamp because as far as I can tell, it's expecting +0800 rather than +08
grok {
match => { "message" => '%{TIMESTAMP_ISO8601:time} %{HOSTNAME:hostname} %{WORD:syslog_program}(?:\[%{NUMBER:syslog_pid}\])?: %{GREEDYDATA:syslog_message}' }
}
If I change the input text from +08 to +0800 it parses it correctly, which is what I'd expect from the github repo with grok patterns
ISO8601_TIMEZONE (?:Z|[+-]%{HOUR}(?::?%{MINUTE}))
But if I check the ISO8601 standards, it looks like +xx is permitted:
So if the time being described is one hour ahead of UTC (such as the time in Berlin during the winter), the zone designator would be "+01:00", "+0100", or simply "+01".
I've worked around this by overwriting the ISO8601_TIMEZONE pattern in a custom pattern for these log files:
ISO8601_TIMEZONE (?:Z|[+-]%{HOUR}(?::?%{MINUTE})?)
The only change I've made between my pattern declaration and the inbuilt one is to make the non-capture group for minutes optional. Is this a bug? Should I be logging this as an issue in github?
Thanks.
EDIT: Also, the date plugin seems to assume timezone will always be 4 digits, I'm not full bottle enough on logstash yet to know if the date plugin uses the same patterns or not (I assume not), but there may be a need (if this a bug that elastic would want fixed) additional code would need adjustment.