Logstash, Date plugin not working as expected

[sguerrero (Sebastian)]

Logstash Version: 7.10.1
OS: Debian 10 Buster
Architecture: x86-64
Kernel: Linux 4.19.0-11-amd64

I'm trying to parse this date:

20210128 94501065

The pattern is:

      date {
         match => ["timestamp", "yyyyMMdd HmmssSSS"]
         target => "@timestamp"
      }

It fails.

          "timestamp" => "20210128 94501065",

"_dateparsefailure"
"@timestamp" => 2021-01-28T15:05:28.283Z,

But I noticed something interesting, if I modify the log to:

20210128 9:45:01.065

So the pattern will be:

      date {
         match => ["timestamp", "yyyyMMdd H:mm:ss.SSS"]
         target => "@timestamp"
      }

It succeed:

          "timestamp" => "20210128 9:45:01.065",
         "@timestamp" => 2021-01-28T12:45:01.065Z,

Date with 2 digits hour works fine:
20210128 104501065

[aaron-nimocks]

Are you able to adjust the log to generate a full hour?

I am not sure how you would get a correct hour with only 1 digit. What would an example timestamp look like from your log for the afternoon hours?

 K       hour of halfday (0~11)       number        0
 h       clockhour of halfday (1~12)  number        12

 H       hour of day (0~23)           number        0
 k       clockhour of day (1~24)      number        24

[sguerrero (Sebastian)]

Thanks for your reply.
Those logs are external I can't modify them.

For example afternoon:
125959416

I'm reading this https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html and it said:

H

hour of the day (24-hour clock)

H
minimal-digit hour. Example: 0 for midnight.
HH
two-digit hour, zero-padded if needed. Example: 00 for midnight.

To my understanding H would match between 0 to 23, but it doesn't if there is not a separator between hours, minutes, seconds and milliseconds (but it does match if I have separatos like H:mm:ss.SSS). So I guess that the problem I'm having is related to the way that the H match the pattern when there is not a separator.

[Badger]

I think the problem is in the underlying Joda library that the date filter uses. When it sees H the builder looks for one or two digits -- it is not limited to one. That will consume '94', which is not a valid hour.

You could use mutate+gsub to change the timestamp to "20210128 9.4501065" and then parse that using "yyyyMMdd H.mmssSSS"

[sguerrero (Sebastian)]

Thank you very much!

I'm doing something like this:

      if [hour_field] =~ /^\d{8}$/ {
         mutate {
            replace => { "hour_field" => "0%{hour_field}" }
         }
      }

I have two fields, first one is the date 20210128, second one is the time, 94501065

yyyyMMdd HHmmssSSS

So far it's working properly.