Parse timestamp using date filter plugin

Elastic Stack Logstash
1

I have a log line:

test-process 2020/06/04 13:35:42 process.go:646: INFO: GetTest: X-Request-Id: 14242dqdq2aA, status: HTTP 200 OK

Below is my filter:

filter {
        dissect {
            mapping => {
                "message" => "%{process} %{timestamp} %{+timestamp} %{function} %{log_level} %{msg}"
            }
            remove_tag => [ "_dissectfailure" ]
        }
        mutate {
            gsub => [
                "message", "\s*\|\s*", "|",
                "log_level", ":$", "" ,
                "function", ":$", ""
            ]
            remove_field => [ "process" ]
        }
        date {
            match => [ "timestamp" , "YYYY/MM/DD HH:mm:ss" ]
            target => "timestamp"
        }
        kv {
            field_split => "|"
        }
}

The problem is with the timestamp in the output. It is "2020-01-04" instead of "2020-06-04" . How can I fix it ?

{
       "message" => "test-process 2020/06/04 13:35:42 process.go:646: INFO: GetTest: X-Request-Id: 14242dqdq2aA, status: HTTP 200 OK",
     "log_level" => "INFO",
           "msg" => "GetTest: X-Request-Id: 14242dqdq2aA, status: HTTP 200 OK",
    "@timestamp" => 2020-06-18T12:53:25.219Z,
      "@version" => "1",
      "function" => "process.go:646",
          "host" => "test.example.com",
     "timestamp" => 2020-01-04T12:35:42.000Z
}
2

DD is day of the year, which overrides month. Use dd, which is day of the month.

1 Like
3

Thanks a lot it fixed the issue

4

Can we also get rid of "@timestamp" => 2020-06-18T12:53:25.219Z," and just use "timestamp" => 2020-06-04T12:35:42.000Z instead ?

5

It is possible to remove @timestamp, but some functionality requires it, so it may break things. For example, sprintf references to a date, such as "logstash-%{+yyyy.MM.dd}" always use @timestamp, so it will resolve to "logstash-". That is just an example, there may be other things.

6

How to make sure that @timestamp uses 020-06-04T12:35:42.000Z instead of 2020-06-18T12:53:25.219Z ?

7

Either remove the target option from the date filter, so that it sets @timestamp, or else use mutate+copy to copy [timestamp] to [@timestamp].

8

Thank you. Let me try this

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.