Date parsing problems

[ta7643]

Hi,

I have a very strange behavior when parsing date. Does anybody understand this behavior?
Here the config:

input { stdin { codec => json} }
filter {
 if [last_seen]{
   date {
    tag_on_failure => true
    match => [ "last_seen", "dd.MM.YYYY HH:mm" ]
    target => "@timestamp_last_seen"
   }
 } 
}
output { stdout { codec => rubydebug } }

And here 3 examples:

{"last_seen": "31.03.2019 01:59"}
{
                    "host" => "example",
                "@version" => "1",
               "last_seen" => "31.03.2019 01:59",
              "@timestamp" => 2019-05-20T13:47:47.855Z,
    "@timestamp_last_seen" => 2019-03-31T00:59:00.000Z
}

{"last_seen": "31.03.2019 02:00"}
{
          "host" => "example",
          "tags" => [
        [0] "true"
    ],
      "@version" => "1",
     "last_seen" => "31.03.2019 02:00",
    "@timestamp" => 2019-05-20T13:48:00.090Z
}

why the above example failed??
If i change the day of the month from 31 to 30 it works again. But 31.03.2019 exists and the expected output should be only 1 minute different from the one in the first example.

{"last_seen": "30.03.2019 02:00"}
{
                    "host" => "example",
                "@version" => "1",
               "last_seen" => "30.03.2019 02:00",
              "@timestamp" => 2019-05-20T13:48:36.551Z,
    "@timestamp_last_seen" => 2019-03-30T01:00:00.000Z
}

Thx in advance

[pastechecker (Charlie)]

Hi,
thanks for the config, very quick problem replication.

{"last_seen": "31.03.2019 01:59"} - works : 2019-03-31T00:59:00.000Z
{"last_seen": "31.03.2019 02:00"} - failed to parse
{"last_seen": "31.03.2019 02:11"} - failed to parse
{"last_seen": "31.03.2019 02:59"} - failed to parse
{"last_seen": "31.03.2019 03:00"} - works : 2019-03-31T01:00:00.000Z

According to the calendar such date exists, but Logstash seems to have 1 hour parsing problem between 2019-03-31T01:00:00.000Z- 2019-03-31T01:59:00.000Z

To be honest it looks like a date parsing corner case.

[Badger]

@ta7643

{"last_seen": "31.03.2019 02:00"}

What date did daylight savings time start in your timezone? Did time go from 01:59:59 to 03:00:00 that day?

[ta7643]

Thats it!.. Thanks a lot

[pastechecker (Charlie)]

Hmm, makes sense.

Is there a way to fix this if we do not control the application that sends such timestamps?
I guess only the condition at the end of your config:

if [tags] =~ /true/{
	mutate {
		gsub => [
		"last_seen", "31.03.2019 02:", "31.03.2019 03:"
		]
	}
}