Date mapping issue - Invalid format ... malformed

Elastic Stack Elasticsearch
1

I have a couple date fields that aren't recognized by ES, and thus causing this Invalid Format issue.

The format is : MM/dd/YYYY (e.g. 11/13/2015)

Here is the error from the ES logs:

    [2015-11-13 13:07:47,830][DEBUG][action.bulk              ] [toph] [hardball-parks][0] failed to execute bulk item (index) index {[hardball-parks][baseballpark][BAL03], source[{"message":["BAL03~Oriole Park I~~Baltimore~MD~05/01/1883~10/10/1889~AA~\"\""],"@version":"1","@timestamp":"2015-11-13T18:07:46.481Z","type":"baseballpark","baseballParkID":"BAL03","parkName":"Oriole Park I","parkAka":null,"parkCity":"Baltimore","parkState":"MD","parkStartDate":"05/01/1883","parkEndDate":"10/10/1889","parkLeague":"AA","parkNotes":""}]}
    MapperParsingException[failed to parse [parkStartDate]]; nested: IllegalArgumentException[Invalid format: "05/01/1883" is malformed at "/01/1883"];
    	at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:339)
...
    Caused by: java.lang.IllegalArgumentException: Invalid format: "05/01/1883" is malformed at "/01/1883"
    	at org.joda.time.format.DateTimeParserBucket.doParseMillis(DateTimeParserBucket.java:187)
    	at org.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:780)
    	at org.elasticsearch.index.mapper.core.DateFieldMapper$DateFieldType.parseStringValue(DateFieldMapper.java:360)
    	at org.elasticsearch.index.mapper.core.DateFieldMapper.innerParseCreateField(DateFieldMapper.java:526)
    	at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:213)
    	at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:331)
    	... 20 more

Here's a sample event from the Logstash rubydebug codec:

{
           "message" => [
        [0] "BAL03~Oriole Park I~~Baltimore~MD~05/01/1883~10/10/1889~AA~\"\""
    ],
          "@version" => "1",
        "@timestamp" => "2015-11-13T17:40:11.734Z",
              "type" => "baseballpark",
    "baseballParkID" => "BAL03",
          "parkName" => "Oriole Park I",
           "parkAka" => nil,
          "parkCity" => "Baltimore",
         "parkState" => "MD",
     "parkStartDate" => "05/01/1883",
       "parkEndDate" => "10/10/1889",
        "parkLeague" => "AA",
         "parkNotes" => ""
}

And finally, here are my ES mappings for this index:

{
  "hardball-parks": {
    "mappings": {
      "baseballPark": {
        "dynamic": "true",
        "include_in_all": false,
        "_all": {
          "enabled": false
        },
        "properties": {
          "baseballParkID": {
            "type": "string",
            "index": "not_analyzed",
            "include_in_all": false
          },
          "parkAka": {
            "type": "string",
            "include_in_all": false
          },
          "parkCity": {
            "type": "string",
            "index": "not_analyzed",
            "include_in_all": false
          },
          "parkEndDate": {
            "type": "date",
            "format": "MM/dd/YYYY",
            "include_in_all": false
          },
          "parkLeague": {
            "type": "string",
            "index": "not_analyzed",
            "include_in_all": false
          },
          "parkName": {
            "type": "string",
            "include_in_all": false
          },
          "parkNotes": {
            "type": "string",
            "include_in_all": false
          },
          "parkStartDate": {
            "type": "date",
            "format": "MM/dd/YYYY",
            "include_in_all": false
          },
          "parkState": {
            "type": "string",
            "index": "not_analyzed",
            "include_in_all": false
          }
        }
      }
    }
  }
}

What am I doing wrong?

3

Don't use old threads. Use your own and eventually link to the old one.

Here:

2016-08-22 13:53:28

is not

yyyy/MM/dd'T'HH:mm:ss
4

Your format doesn't conform with Built-in formats of Elastic Search 2.3! Please refer 'Built In Formats' under https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-date-format.html

I did face exactly the same issue! But I tweaked my PHP time object's format to be in accordance with Elastic Search Mapping!

Curl command to create mapping for the the field 'ClickTime' in a standard format :

curl -XPUT localhost:9200/telemetry/_mapping/DashBoardVisits?
pretty -d "{"properties" : {"ClickTime" : {"type" :"date" , "format" :
"yyyy-MM-dd HH:mm:ss" }}}"
PHP code to create a time object matching the type and format of 'ClickTime' in Elastic Search :

$curtime = date_create();
$kibanadate = date_format($curtime,"Y-m-d H:i:s");

5

I have made necessary changes. Can we please delete the comment? This is no more valid now.