Hi,
I am at a loss, we have to deal with german encodings of month such as Dezember instead of December, or März instead of March, which also impacts the MMM notation instead of Mar it is Mär. This is an example message:
23 Mär 2021 08:28:08,789 INFO additional log message
This is my custom grok pattern for the datetime JBOSSSERVERLOG %{MONTHDAY} %{MONTH} %{YEAR} %{TIME}
This is my logstash config:
input {
file {
path => "/Users/philipp/Downloads/logstash/log/demolog.log"
}
}
filter {
grok {
patterns_dir => ["/Users/philipp/Downloads/logstash/pipeline/custompattern"]
match => [
"message","%{JBOSSSERVERLOG:timestamp} %{LOGLEVEL:log.level} %{GREEDYDATA:message}"
]
add_field => ["received_at", "%{@timestamp}"]
}
if [timestamp] =~ /(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|June?|July?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)/ {
date {
tag_on_failure => ["english-datetime-error"]
match => [
"timestamp",
"dd MMM yyyy HH:mm:ss,SSS",
"MMM dd, yyyy h:mm:ss a",
"MMM dd, yyyy hh:mm:ss a",
"MMM dd, yyyy hh:mm:ss,SSS a",
"dd/MMM/yyyy:HH:mm:ss Z"
]
}
}else{
date {
locale => "de-AT"
tag_on_failure => ["german-datetime-error"]
match => [
"timestamp",
"dd MMM yyyy HH:mm:ss,SSS"
]
}
}
}
output {
stdout{
codec => json
}
}
However it always ends up in something like this
{
"@timestamp": "2021-03-23T12:15:14.993Z",
"host": "TAG-499.local",
"log.level": "INFO",
"path": "/Users/philipp/Downloads/logstash/log/demolog.log",
"message": [
"23 Mär 2021 08:28:08,789 INFO additional log message",
"additional log message"
],
"timestamp": "23 Mär 2021 08:28:08,789",
"tags": ["german-datetime-error"],
"@version": "1",
"received_at": "2021-03-23T12:15:14.993Z"
}
Which makes no sense, since the date
filter from logstash should match my own created timestamp field dd MMM yyyy HH:mm:ss,SSS
.
I tried using de
,de-AT
,de-DE
as locale but none of them worked. If I remove the tag on failure I get the default error with _dateparsefailure
.
Here is a gist of running logstash with the --debug
option. logstash datetime errors · GitHub
Any idea where I am going wrong?