Logstash, Date plugin not working as expected

Logstash Version: 7.10.1
OS: Debian 10 Buster
Architecture: x86-64
Kernel: Linux 4.19.0-11-amd64

I'm trying to parse this date:

20210128 94501065

The pattern is:

      date {
         match => ["timestamp", "yyyyMMdd HmmssSSS"]
         target => "@timestamp"

It fails.

          "timestamp" => "20210128 94501065",

"@timestamp" => 2021-01-28T15:05:28.283Z,

But I noticed something interesting, if I modify the log to:

20210128 9:45:01.065

So the pattern will be:

      date {
         match => ["timestamp", "yyyyMMdd H:mm:ss.SSS"]
         target => "@timestamp"

It succeed:

          "timestamp" => "20210128 9:45:01.065",
         "@timestamp" => 2021-01-28T12:45:01.065Z,

Date with 2 digits hour works fine:
20210128 104501065

Are you able to adjust the log to generate a full hour?

I am not sure how you would get a correct hour with only 1 digit. What would an example timestamp look like from your log for the afternoon hours?

 K       hour of halfday (0~11)       number        0
 h       clockhour of halfday (1~12)  number        12

 H       hour of day (0~23)           number        0
 k       clockhour of day (1~24)      number        24

Thanks for your reply.
Those logs are external I can't modify them.

For example afternoon:

I'm reading this https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html and it said:


hour of the day (24-hour clock)

minimal-digit hour. Example: 0 for midnight.
two-digit hour, zero-padded if needed. Example: 00 for midnight.

To my understanding H would match between 0 to 23, but it doesn't if there is not a separator between hours, minutes, seconds and milliseconds (but it does match if I have separatos like H:mm:ss.SSS). So I guess that the problem I'm having is related to the way that the H match the pattern when there is not a separator.

I think the problem is in the underlying Joda library that the date filter uses. When it sees H the builder looks for one or two digits -- it is not limited to one. That will consume '94', which is not a valid hour.

You could use mutate+gsub to change the timestamp to "20210128 9.4501065" and then parse that using "yyyyMMdd H.mmssSSS"

1 Like

Thank you very much!

I'm doing something like this:

      if [hour_field] =~ /^\d{8}$/ {
         mutate {
            replace => { "hour_field" => "0%{hour_field}" }

I have two fields, first one is the date 20210128, second one is the time, 94501065

yyyyMMdd HHmmssSSS

So far it's working properly.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.