Date filter problems after upgrade

[Renato]

Hey,

we are having problems after we upgraded elk to 7.12.1 (I'm not sure which version we previously used 7.xx). The date filter which we previously used is not working anymore. We don't get any logs in elastic. If I write the wrong date filter I get dateparsefailure but at least I can see the logs in elastic.

 filter {
    grok {
       break_on_match => false
        match => { "message" => "%{TIMESTAMP_ISO8601:encodertime}] %{GREEDYDATA:errorlog}" }
        match => { "[log][file][path]" => "%{URIPATH}/channel\-%{WORD:channel}\.log" }
        }


   date{
       match => [ "encodertime", "YYYY-MM-dd HH:mm:ss" ]
       target => "@timestamp"
#       timezone => "UTC"
        }
}

My log:
[2021-06-03 04:51:44] [@ 0x6764688aff0343656100] error message 2386751. Check manual for additional information.

I can see that grok is working and also there is nothing useful in logstash logs. I also tried with DATESTAMP which parses the logs but it is still not working.

Any help would be highly appreciated.

[Dennis1 (Dennis)]

Hi @Renato, I had the exact same problem and after hours of troubleshooting figured it out!

You need to change your encodertime syntax to "yyyy-MM-dd HH:mm:ss" (i.e., you must use lower case "yyyy"). Apparently in the update, the allowance of capital "YYYY" must have been removed. This should definitely be reported as a Major Breaking Change/Bug!!!

In my case, I was using the date filter as part of an elastic ingestion pipeline, but am assuming the same thing will affect your case with logstash too.

We ruined a lot of log data due to this! To clarify what actually happens when you use capital YYYY instead, is the date gets processed in some bizarre way that--at least for me--changed it to the first few days of January for the specified year...e.g., provided the date "2021-06-24" would be converted to "2021-01-04". I have no idea how or why that happened, and probably relates to some bug in the source code.