Date parse failure, cannot figure out pattern for literal characters in time stamp field

[tomj (Tom Johnson)]

Using Logstash 6.4.3

I am processing Kinesis records from the Kinesis input plugin, where the 'timestamp' field looks like this:

2019-05-02T16:25:17.850884549Z

I am using the following date filter syntax (following suggestions in the docs):

filter { date { match => ["timestamp", "yyyy-MM-dd'T'HH:mm:ss,SSS'Z'"] target => "timestamp" } }

However, Logstash does not accept this; here is an example log message (extraneous fields omitted):

[2019-05-02T17:06:59,934][DEBUG][logstash.pipeline ] output received {"event"=>{"@version"=>"1", "fields"=>{ "@timestamp"=>2019-05-02T17:06:59.833Z, "timestamp"=>2019-05-02T17:06:57.880Z, "tags"=>["_dateparsefailure"]}}

Edit: I should also mention that I get similar results when using the ISO8601 format literal.

[Badger]

@tomj

"timestamp"=>2019-05-02T17:06:57.880Z

There are no quotes around timestamp, so that is a Logstash::TimeStamp, not a string. A date filter cannot parse a Logstash::Timestamp.

I am not familiar with kinesis but I know that for jdbc the input will convert datetimes in the database into Logstash::Timestamp for you, so you do not need a date filter. It may be the same for kinesis.

If adding

 mutate { convert  {"timestamp" => "string"} }

makes the error go away then you can just delete the date filter.

[tomj (Tom Johnson)]

I apologize, I should have stated I was using the JSON codec with the Kinesis input for clarity.

As it turns out, I don't need either the date filter or the mutate to make it work. I inherited the configuration and just assumed that it was working all along. I had never checked the output and thus did not see the _dateparsefailure messages until now.

Thank you, and thanks for the link to the issue. I searched Logstash issues but not the specific plugin repository issues; now I know better.