Elatic mapper_parsing_exception

[savvy]

[2020-05-06T17:16:33,007][WARN ][logstash.outputs.elasticsearch][mykafka] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"mydata-day-2020.05.06-113", :routing=>nil, :_type=>"_doc"}, #LogStash::Event:0x4f163538], :response=>{"index"=>{"_index"=>"mydata-day-2020.05.06-113", "_type"=>"_doc", "_id"=>"-s_Y63EBQO9FwNtA5t0Z", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [abc.timestamp] of type [date_nanos] in document with id '-s_Y63EBQO9FwNtA5t0Z'. Preview of field's value: ''2020-05-06T01:46:37.357884414Z''", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"failed to parse date field ['2020-05-06T01:46:37.357884414Z'] with format [yyyy-MM-dd HH:mm:ss.SSSSSSSSS Z || yyyy-MM-dd'T'HH:mm:ss.SSSSSSSSSZ]", "caused_by"=>{"type"=>"date_time_parse_exception", "reason"=>"date_time_parse_exception: Failed to parse with all enclosed parsers"}}}}}}

Using elastic 7.5.1 - Using below template
{
"order": 2,
"index_patterns": [
"mydata*"
],
"settings": {
"index": {
"refresh_interval": "5s"
}
},
"mappings": {
"properties": {
"abc.timestamp": {
"format": "yyyy-MM-dd HH:mm:ss.SSSSSSSSS Z || yyyy-MM-dd'T'HH:mm:ss.SSSSSSSSSZ",
"type": "date_nanos"
}
}
},
"aliases": {}
}

What format should I be using?

[Luca_Belluccini]

Hello @savvy

On 7.x, the date formats must be converted to Java Time as detailed here

You can use the following tool to convert the formats from Joda to Java time: https://esddd.herokuapp.com/

DELETE timenanos
PUT timenanos
{
  "settings": {
    "index": {
      "refresh_interval": "5s"
    }
  },
  "mappings": {
    "properties": {
      "abc": {
        "properties": {
          "timestamp": {
            "format": "uuuu-MM-dd HH:mm:ss.SSSSSSSSS X||yyyy-MM-dd'T'HH:mm:ss.SSSSSSSSSX",
            "type": "date_nanos"
          }
        }
      }
    }
  }
}

POST timenanos/_doc/1
{
  "abc": {
    "timestamp": "2020-05-06T01:46:37.357884414Z"
  }
}
POST timenanos/_doc/2
{
  "abc": {
    "timestamp": "2020-05-06 01:46:37.357884414 Z"
  }
}