Unable to parse datetime from log message

Elastic Stack Logstash
1

I'm unable to extract the datetime field from the log streamed from filebeat to elsaticsearch via logstash. I'm using date filter to extract the timestamp from the log message and set it to @timestamp. Since I already spent few hours without any luck I'm reaching out to for help.

Thanks in advance.

Here's the info:

logstash output:

[0] "beats_input_codec_plain_applied",
[1] "_dateparsefailure"
],
"message" => "[2019-01-08 01:49:04] [INFO ] [test] Test log message ",

Date filter:
filter {
date {
match => [ "message", "YYYY-mm-dd HH:mm:ss" ]
target => "@timestamp"
}
}

2

The pattern in the date filter has to match the whole of the field that you pass to it. You can use dissect to extract the timestamp from the message. Also, note than months are MM, not mm.

Try

    dissect { mapping => { "message" => "[%{ts} %{+ts}]%{}" } }
    date { match => [ "ts", "YYYY-MM-dd HH:mm:ss" ] }
3

Thanks, your solution worked. However, I don't see the log statements with matched timestamp entries from Kibana UI. UI is only showing unmatched entries. I do see the updated @timestamp field in console output. I'm using following logstash filter. Pardon my errors as I'm going thru the learning process.

filter {
    dissect { mapping => { "message" => "[%{ts} %{+ts}]%{}" } }
    date {
      match => [ "ts", "YYYY-MM-dd HH:mm:ss" ]
      target => "@timestamp"
    }
}
4

If the date filter does not match then @timestamp on the events will be current and they will show up in a "Last 15 minutes" view. If the date filter does match then you may need a "Last month" view to see them.

5

Works like a charm. Thank You!

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.