Add date and seconds to odd(?) ISO 8601-ish timestamp?

[tjswe]

Im logging AIS-data (ship positions) into Elasticsearch.

The "timestamp" field from the receiver is 99% of the time a regular ISO8601 timestamp like "2021-10-29T17:55:30Z" which logstash takes in without issues. But some weatherbouys are sending the timestamp as "29T18:25Z" which Logstash does not like (=>"failed to parse date field [29T18:25Z] with format [strict_date_optional_time||epoch_millis])

Can i modify 29T18:25Z to become 2021-10-29T18:25:00Z in Logstash (or in some other way make Elastic accept it)? I have tried searching and reading without luck. Anyone have a solution or a pointer on what component to use?

[Badger]

My initial thought was

    mutate { add_field => { "[ts]" => "29T18:25Z" } }
    date { match => [ "ts", "dd'T'HH:mm'Z'" ] timezone => "UTC" target => "[@metadata][ts]" }
    mutate {
        convert => { "[@metadata][ts]" => "string" }
        gsub => [ "[@metadata][ts]", "-01-", "-%{+MM}-" ]
        add_field => { "ts2" => "%{[@metadata][ts]}" }
    }

which will get you

       "ts2" => "2021-10-29T18:25:00.000Z",
        "ts" => "29T18:25Z"

However, the date filter is willing to guess the year (because some syslog formats do not include the year in the timestamp), but it is not willing to guess the month and defaults it to 1. That means that same year guessing code will guess wrong for the whole of December.

If you just want to add today's year and month you could try

    mutate { add_field => { "ts3" => "%{+YYYY-MM-}%{ts}" } }
    mutate { gsub => [ "ts3", "Z$", ":00Z" ] }

which produces

       "ts3" => "2021-10-29T18:25:00Z",

Note that whatever heuristics you use to guess the month and year, it will be a guess and it will sometimes go wrong. For example, if 31T23:59z is sent on the 31st of January and it gets delayed and processed after midnight it will not even be a valid date, since February never has 31 days.

[tjswe]

Many thanks for that comprehensive answer! Will look into it!