Add date and seconds to odd(?) ISO 8601-ish timestamp?

Elastic Stack Logstash
2

My initial thought was

    mutate { add_field => { "[ts]" => "29T18:25Z" } }
    date { match => [ "ts", "dd'T'HH:mm'Z'" ] timezone => "UTC" target => "[@metadata][ts]" }
    mutate {
        convert => { "[@metadata][ts]" => "string" }
        gsub => [ "[@metadata][ts]", "-01-", "-%{+MM}-" ]
        add_field => { "ts2" => "%{[@metadata][ts]}" }
    }

which will get you

       "ts2" => "2021-10-29T18:25:00.000Z",
        "ts" => "29T18:25Z"

However, the date filter is willing to guess the year (because some syslog formats do not include the year in the timestamp), but it is not willing to guess the month and defaults it to 1. That means that same year guessing code will guess wrong for the whole of December.

If you just want to add today's year and month you could try

    mutate { add_field => { "ts3" => "%{+YYYY-MM-}%{ts}" } }
    mutate { gsub => [ "ts3", "Z$", ":00Z" ] }

which produces

       "ts3" => "2021-10-29T18:25:00Z",

Note that whatever heuristics you use to guess the month and year, it will be a guess and it will sometimes go wrong. For example, if 31T23:59z is sent on the 31st of January and it gets delayed and processed after midnight it will not even be a valid date, since February never has 31 days.

2 Likes