Datetime parsing errors when including locale de-AT

[philippkahr]

Hi,

I am at a loss, we have to deal with german encodings of month such as Dezember instead of December, or März instead of March, which also impacts the MMM notation instead of Mar it is Mär. This is an example message:

23 Mär 2021 08:28:08,789 INFO additional log message

This is my custom grok pattern for the datetime JBOSSSERVERLOG %{MONTHDAY} %{MONTH} %{YEAR} %{TIME}

This is my logstash config:

input {
  file {
    path => "/Users/philipp/Downloads/logstash/log/demolog.log"
  }
}

filter {
    grok {
      patterns_dir => ["/Users/philipp/Downloads/logstash/pipeline/custompattern"]
      match => [
        "message","%{JBOSSSERVERLOG:timestamp} %{LOGLEVEL:log.level} %{GREEDYDATA:message}"
        ]
      add_field => ["received_at", "%{@timestamp}"]
    }
    if [timestamp] =~ /(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|June?|July?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)/ {
      date {
        tag_on_failure => ["english-datetime-error"]
        match => [
          "timestamp",
          "dd MMM yyyy HH:mm:ss,SSS",
          "MMM dd, yyyy h:mm:ss a",
          "MMM dd, yyyy hh:mm:ss a",
          "MMM dd, yyyy hh:mm:ss,SSS a",
		      "dd/MMM/yyyy:HH:mm:ss Z"
          ]
      }
    }else{
      date {
        locale => "de-AT"
        tag_on_failure => ["german-datetime-error"]
        match => [
          "timestamp",
          "dd MMM yyyy HH:mm:ss,SSS"
          ]
      }
    }
}

output {
    stdout{
      codec => json 
      }
}

However it always ends up in something like this

{
  "@timestamp": "2021-03-23T12:15:14.993Z",
  "host": "TAG-499.local",
  "log.level": "INFO",
  "path": "/Users/philipp/Downloads/logstash/log/demolog.log",
  "message": [
    "23 Mär 2021 08:28:08,789 INFO additional log message",
    "additional log message"
  ],
  "timestamp": "23 Mär 2021 08:28:08,789",
  "tags": ["german-datetime-error"],
  "@version": "1",
  "received_at": "2021-03-23T12:15:14.993Z"
}

Which makes no sense, since the date filter from logstash should match my own created timestamp field dd MMM yyyy HH:mm:ss,SSS.

I tried using de,de-AT,de-DE as locale but none of them worked. If I remove the tag on failure I get the default error with _dateparsefailure.

Here is a gist of running logstash with the --debug option. logstash datetime errors · GitHub

Any idea where I am going wrong?