Date Mapping Parse Failure after upgrade to 7.1.1

[johnwood]

We have just upgraded to 7.1.1 (from 6.7) on the Elastic Cloud service.

Our previously happy logstash is now erroring out with a date related parsing issue when trying to put a date field into ES.

The error we get is:

 [2019-06-07T17:22:55,068][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"pnm-2019.06.07", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x1096275e>], :response=>{"index"=>{"_index"=>"pnm-2019.06.07", "_type"=>"_doc", "_id"=>"58fRMGsBjCggiYigqPIl", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [AgentTime] of type [date] in document with id '58fRMGsBjCggiYigqPIl'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"failed to parse date field [2019.06.07 17:22:54.626 GMT+10:00] with format [yyyy.MM.dd HH:mm:ss.SSS 'GMT'Z]", "caused_by"=>{"type"=>"date_time_parse_exception", "reason"=>"Text '2019.06.07 17:22:54.626 GMT+10:00' could not be parsed at index 27"}}}}}}

In short we are trying to get this date into ES: "2019.06.07 17:22:54.626 GMT+10:00"

The mapping in ES looks like this:
"AgentTime": {
"type": "date",
"format": "yyyy.MM.dd HH:mm:ss.SSS 'GMT'Z"
},

Any advice or pointers would be appreciated

Thanks!

[dadoonet (David Pilato)]

I'm pretty sure @spinscale can help.

[spinscale (Alexander Reelsen)]

Try this as a format: "yyyy.MM.dd HH:mm:ss.SSS 'GMT'XXX" under 7.x

Your format should have been logged as a deprecation since 6.7, but has only recently been added to the 6.8 branch. See this PR

hope this helps!

[johnwood]

Wow - thanks so much, just updated our mapping and 'boink' we are indexing again. Thanks so much - if you get to Sydney I owe you a beer!