Version: logstash 5.6.3
We have 2 date filters in our logstash config.
The first parses the timestamp field we matched in our logs. We want to have always the same timestamp format so we use date to do this.
Second date is used to set @timestamp to timestamp for each event, if we have a timestamp parsed. But this fails if we altered date with timestamp.
I created a little test case to reproduce the issue.
input { stdin {} }
filter {
grok {
match => ["message", "%{TIMESTAMP_ISO8601:timestamp} %{WORD}"]
}
# trying to fix timestamp parsing
date {
match => [ "timestamp", "ISO8601", "UNIX", "UNIX_MS", "TAI64N", "yyyy-MM-dd HH:mm:ss", "MMM dd YYYY HH:mm:ss", "YYYY/MM/dd HH:mm:ss", "dd/MMM/YYYY:HH:mm:ss Z" ]
target => "timestamp"
}
# if we have a valid timestamp, set is as @timestamp instead of parsed time of logstash
date {
match => [ "timestamp", "ISO8601"]
target => "@timestamp"
}}
output { stdout { codec => rubydebug } }
# echo '2011-04-19T03:44:01.103Z blabla' | /usr/share/logstash/bin/logstash -f test_logstash.config
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
{
"@version" => "1",
"host" => "test-host",
"@timestamp" => 2017-10-25T10:06:16.527Z,
"message" => "2011-04-19T03:44:01.103Z blabla",
"timestamp" => 2011-04-19T03:44:01.103Z,
"tags" => [
[0] "_dateparsefailure"
]
}