Logstash -- inconsistent result with Date

  • Version: 5.6.3
  • Operating System: RedHat 7.4
  • Sample Data: "2019-03-10 02:00:00" will not work.
  • Steps to Reproduce:

create a file with the following value:

"2019-03-10 02:00:00"

Here is a Logstash config. file::

input {
beats {
port => "5046"
}
}

filter {
csv {

columns => ["UsageEndDate"]
separator => ","

}

date {
match => ["UsageEndDate", "yyyy-MM-dd HH:mm:ss"]
timezone => "America/New_York"
target => "newEndDate"
}

}

output {
stdout { codec => rubydebug }

}

==========================================
After kick off logstash, I am keep gettting _dateparsefailure:

{
"@timestamp" => 2019-04-08T18:08:49.064Z,
"offset" => 23,
"@Version" => "1",
"input_type" => "log",
"beat" => {
"name" => "kibana",
"hostname" => "kibana",
"version" => "5.6.3"
},
"host" => "kibana",
"UsageEndDate" => "2019-03-10 02:00:00",
"source" => "/mypath/aws-nc2.test",
"message" => ""2019-03-10 02:00:00"",
"type" => "log",
"fields" => {
"index" => "testing_aws"
},
"tags" => [
[0] "beats_input_codec_plain_applied",
[1] "_dateparsefailure"
]
}

I had no issue if the data is 2019-03-10 01:00:00

or any other hours, but only had issue with 02:00:00.

Thanks,

Noah

Daylight savings started on 3/10. There was no 2 AM. We went straight from 01:59:59 to 03:00:00.

Thanks so much for quick reply. Any recommendation, other than looking for all my data and change them from 2 to 3?

That's what I did when I was looking at NYPD arrest data, which had this issue. I had a function that would look for the start of EDT in each year and fix any affected timestamps.

1 Like

Ok, thanks. I just fixed issue by changing the timezone from timezone =>"America/New_York" to timezone='EST'

Not sure why it solved that if both were the same.

EST and EDT are two different timezones. EST is 5 hours behind GMT. EDT is 4 hours behind. America/New_York uses EDT from March to November.

ok, so I am working with AWS billing file. I went to confirm what timezone they were using for the data. Found out it was UTC. I fixed the issue by, using timezone=> "UTC".

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.