Ingesting both 'standard' timestamp syslog and ISO8601 timestamped syslog

(Nathan Harper) #1

As per we are happily ingesting Filebeat > logstash > elasticsearch for system logs that use the 'standard' syslog timestamp (eg May 16 13:58:17). However we have some systems that output in ISO8601 format (2018-05-16T14:06:23.106124+01:00).

Any suggestions on how I can modify the reference config to be able to filter both?


The date filter match option takes an array, which can include multiple date formats to try.

(Nathan Harper) #3

ok, so I could replace:

match => { "message" => ["%{SYSLOGTIMESTAMP:[system][syslog][timestamp]} %{SYSLOGHOST:[system][syslog][hostname]} %{DATA:[system][syslog][program]}(?:[%{POSINT:[system][syslog][pid]}])?: %{GREEDYMULTILINE:[system][syslog][message]}"] }

with match => { "message" => ["%{SYSLOGTIMESTAMP:[system][syslog][timestamp]} %{SYSLOGHOST:[system][syslog][hostname]} %{DATA:[system][syslog][program]}(?:[%{POSINT:[system][syslog][pid]}])?: %{GREEDYMULTILINE:[system][syslog][message]}, %{TIMESTAMP_ISO8601:[system][syslog][timestamp]} %{SYSLOGHOST:[system][syslog][hostname]} %{DATA:[system][syslog][program]}(?:[%{POSINT:[system][syslog][pid]}])?: %{GREEDYMULTILINE:[system][syslog][message]}"] }

and that should do the trick?


Oh, yes, grok takes an array as well, so you can match both patterns against the line, as well as matching both date formats against [system][syslog][timestamp] in the date filter.

(Nathan Harper) #5

ok, great, I'll investigate further! Thanks for the help.

(system) #6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.