Im logging AIS-data (ship positions) into Elasticsearch.
The "timestamp" field from the receiver is 99% of the time a regular ISO8601 timestamp like "2021-10-29T17:55:30Z" which logstash takes in without issues. But some weatherbouys are sending the timestamp as "29T18:25Z" which Logstash does not like (=>"failed to parse date field [29T18:25Z] with format [strict_date_optional_time||epoch_millis])
Can i modify 29T18:25Z to become 2021-10-29T18:25:00Z in Logstash (or in some other way make Elastic accept it)? I have tried searching and reading without luck. Anyone have a solution or a pointer on what component to use?
However, the date filter is willing to guess the year (because some syslog formats do not include the year in the timestamp), but it is not willing to guess the month and defaults it to 1. That means that same year guessing code will guess wrong for the whole of December.
If you just want to add today's year and month you could try
Note that whatever heuristics you use to guess the month and year, it will be a guess and it will sometimes go wrong. For example, if 31T23:59z is sent on the 31st of January and it gets delayed and processed after midnight it will not even be a valid date, since February never has 31 days.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.