Hello,
I figured out one thing:
Logstash 7.6.0 is not able map date to @timestamp field when timestamp has 02 in hour positio.
In pipeline configuration file:
if [timestamp] {
date {
match => ["timestamp", "yyyy-MM-dd HH:mm:ss,SSS", "yyyy-MM-dd HH:mm:ss.SSS", "ISO8601" ]
target => "@timestamp"
remove_field => ["timestamp"]
}
}
Grok pattern works fine. In the field timestamp is correct value, e.g.: 2020-03-29 03:35:47.392 without additional whitespaces.
I simplified parser and date mapping and found out that error _dateparsefailure occur only on lines where hour is 02.
2020-03-29 02:35:47.392 INFO [xxx] Some message
There is example of log file.
2020-03-29 15:35:47.392 INFO [xxx] Some message
2020-03-29 14:35:47.392 INFO [xxx] Some message
2020-03-29 13:35:47.392 INFO [xxx] Some message
2020-03-29 12:35:47.392 INFO [xxx] Some message
2020-03-29 11:35:47.392 INFO [xxx] Some message
2020-03-29 10:35:47.392 INFO [xxx] Some message
2020-03-29 09:35:47.392 INFO [xxx] Some message
2020-03-29 08:35:47.392 INFO [xxx] Some message
2020-03-29 07:35:47.392 INFO [xxx] Some message
2020-03-29 06:35:47.392 INFO [xxx] Some message
2020-03-29 05:35:47.392 INFO [xxx] Some message
2020-03-29 04:35:47.392 INFO [xxx] Some message
2020-03-29 03:35:47.392 INFO [xxx] Some message
2020-03-29 02:35:47.392 INFO [xxx] Some message
2020-03-29 01:35:47.392 INFO [xxx] Some message
Aaa.. so it looks that problem is in daylight saving time change. Yes, the time should go to 03:00:00. So how we can achieve to correctly map this dates?
That only deals with the one hour. When your logs contain "2020-03-29 03:35:47.392" they may well mean "2020-03-29 04:35:47.392". It also ignores the problems when time goes back later in the year. Personally I only cared whether arrests were recorded in the right month, so an hour with twice as many events as it should have had, and an hour that had no events did not bother me.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.