Ingesting both 'standard' timestamp syslog and ISO8601 timestamped syslog

Elastic Stack Logstash
1

As per https://www.elastic.co/guide/en/logstash/current/logstash-config-for-filebeat-modules.html we are happily ingesting Filebeat > logstash > elasticsearch for system logs that use the 'standard' syslog timestamp (eg May 16 13:58:17). However we have some systems that output in ISO8601 format (2018-05-16T14:06:23.106124+01:00).

Any suggestions on how I can modify the reference config to be able to filter both?

2

The date filter match option takes an array, which can include multiple date formats to try.

3

ok, so I could replace:

match => { "message" => ["%{SYSLOGTIMESTAMP:[system][syslog][timestamp]} %{SYSLOGHOST:[system][syslog][hostname]} %{DATA:[system][syslog][program]}(?:[%{POSINT:[system][syslog][pid]}])?: %{GREEDYMULTILINE:[system][syslog][message]}"] }

with match => { "message" => ["%{SYSLOGTIMESTAMP:[system][syslog][timestamp]} %{SYSLOGHOST:[system][syslog][hostname]} %{DATA:[system][syslog][program]}(?:[%{POSINT:[system][syslog][pid]}])?: %{GREEDYMULTILINE:[system][syslog][message]}, %{TIMESTAMP_ISO8601:[system][syslog][timestamp]} %{SYSLOGHOST:[system][syslog][hostname]} %{DATA:[system][syslog][program]}(?:[%{POSINT:[system][syslog][pid]}])?: %{GREEDYMULTILINE:[system][syslog][message]}"] }

and that should do the trick?

4

Oh, yes, grok takes an array as well, so you can match both patterns against the line, as well as matching both date formats against [system][syslog][timestamp] in the date filter.

5

ok, great, I'll investigate further! Thanks for the help.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.