Date filter year not parsing correctly

Elastic Stack Logstash
1

This is my sample log line

2021-06-28 10:25:29.537695 traceID=d283c222257e0e92ba97269a5f780d81 spanID=f46645e821cb4dab A4 LG 4 Logger.cpp:250 - Configuring log file: /home/rxm/log/latest/2021-06-28-102529-log-%3N.log Archival path: /home/rxm/log/latest/

2021-06-28 10:25:29.537837 traceID=d283c222257e0e92ba97269a5f780d81 spanID=f46645e821cb4dab A4 LG 5 RxBaseNode.cpp:4475 - Log root folder: ---------- /home/rxm/log/latest/

2021-06-28 10:25:29.537861 traceID=d283c222257e0e92ba97269a5f780d81 spanID=f46645e821cb4dab A4 LG 6 RxBaseNode.cpp:450 - Configuration root folder:- /home/rxm/config/

And this is my Config.

filter {
  grok {
    match => {
      "message" => "%{DATESTAMP:logdate} traceID=%{WORD:traceID} spanID=%{WORD:spanID} %{GREEDYDATA:msg}"
    }
  }
  date {
    match => [ "logdate" , "yyyy-MM-dd HH:mm:ss.SSSSSS" ]
    target => "@timestamp"
  }
}

But the date filter is parsing the log date incorrectly as

2022-02-10T17:48:36.964Z

Here is the output

{
      "@version" => "1",
    "@timestamp" => 0021-06-28T04:32:01.537Z,
          "host" => "LTM-PKRISHNAN.Dlink",
          "type" => "newapiserver",
        "spanID" => "f46645e821cb4dab",
       "message" => "2021-06-28 10:25:29.537 traceID=d283c222257e0e92ba97269a5f780d81 spanID=f46645e821cb4dab A4 LG 6 RxBaseNode.cpp:450 - Configuration root folder:- /home/rxm/config/",
          "path" => "/Users/pkrishnan/Node-Apps/my-node-example/logs/newapiserver.log",
       "logdate" => "21-06-28 10:25:29.537",
           "msg" => "A4 LG 6 RxBaseNode.cpp:450 - Configuration root folder:- /home/rxm/config/",
       "traceID" => "d283c222257e0e92ba97269a5f780d81"
}

Why is the timestamp having a date in which I get 0021 for the year instead of 2021. This is puzzling

2

The error is in your grok, the output of your logdate field is wrong, your year only have two digits.:

"logdate" => "21-06-28 10:25:29.537"

The DATESTAMP pattern is not the correct one for the date format you have, you shoud use the TIMESTAMP_ISO8601 pattern, this will correctly parse your date.

You can check the patterns here.

3

As Leandro says, this is the wrong pattern. It matches DATE_EU or DATE_US. In this case it will match DATE_EU - 21-06-28, which is 21st June 1928. If you had anchored your pattern ^%{DATESTAMP:logdate} (which is always a good thing) then the grok would have failed to match.

2 Likes
4

@leandrojmp @Badger
Thank you. It parses fine now to this:

{
        "spanID" => "f46645e821cb4dab",
          "host" => "LTM-PKRISHNAN.Dlink",
       "traceID" => "d283c222257e0e92ba97269a5f780d81",
      "@version" => "1",
       "message" => "2021-06-28 10:25:29.537911 traceID=d283c222257e0e92ba97269a5f780d81 spanID=f46645e821cb4dab A4 LG 9 Parameter.cpp:55 - Parameter group systemConfig successfully registered.",
          "type" => "newapiserver",
           "msg" => "A4 LG 9 Parameter.cpp:55 - Parameter group systemConfig successfully registered.",
    "@timestamp" => 2021-06-28T04:55:29.537Z,
          "path" => "/Users/pkrishnan/Node-Apps/my-node-example/logs/newapiserver.log",
       "logdate" => "2021-06-28 10:25:29.537911"
}

The difference in time is due to 'timezone' and I can add that. Thank you!
1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.