Match timestamps using different locales

[friesoft (Bernhard Friedreich)]

Hi,

we have some logfiles which contain three different timestamp formats.
Example:

[02/Mar/2018:09:35:21] [Info] [Module] [3085] [Function] my message
[Fri Mar 02 09:35:21.474812 2018] [Module] [pid 1819:tid 140437688055616] errorcode: caught SIGTERM, shutting down
[02/Mär/2018:09:35:25] [Info] [Module] [19154] [Function: message]

The timestamp is always contained in the field "timestamp", extracted using a grok pattern.

I've tried the following:

     date {
         match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss" ] # 02/Mar/2018:09:35:21
         locale => [ "en-US" ]
     }
     date {
         match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss" ] # 02/Mär/2018:09:35:25
         locale => [ "de-DE" ]
     }
     date {
         match => [ "timestamp", "EEE MMM dd HH:mm:ss.SSSSSS yyyy" ] # Fri Mar 02 09:35:30.052734 2018
     }

The problem is: I get dateparsefailure on all three lines. How to get rid of those? The timestamps themselves are correctly parsed.

What would be the correct way to handle such special logfiles?..

Thanks
Bernhard

[magnusbaeck (Magnus Bäck)]

You can clear the date filter's tag_on_failure option to avoid the _dateparsefailure tag. If you want to keep that tag if none of the filters matched that's possible but will require a bunch of conditionals and probably a few mutate filters.

[friesoft (Bernhard Friedreich)]

Thanks for the info! didn't grasp that tag_on_failure was the right thing for me
Works for me now.

I just noticed one of the patterns is still wrong - the time contains microseconds - which it looks like can't be parsed using logstash/jodatime.
What can I do to match the rest of the date/time?

Fri Mar 02 10:03:03.048488 2018

     date {
         match => [ "timestamp", "EEE MMM dd HH:mm:ss.SSS**SSS** yyyy" ]
         tag_on_failure => [ ]
     }

Thanks for the fast reply

[magnusbaeck (Magnus Bäck)]

You might have to use a mutate filter's gsub option to remove the microseconds. The date filter doesn't keep more than millisecond precision anyway.