ES 7.0 New Date Format

[ben-charles]

We are currently upgrading from ES 6.8.2 to 7.0.0. We have date fields in our documents that are stored with the format
yyyy-MM-dd'T'HH:mm:ss.SSSZ

However, since in ES 7.0 uses java.util.time instead of Joda based time package, I keep getting an error that ElasticSearch is unable to parse this field.

Caused by: ElasticsearchParseException[failed to parse date field [1970-01-01T00:00:00.003Z] with format [epoch_millis||date_time||date_time_no_millis]:

Based on this answer from a previous discussion, we should now use XXX instead of Z for timezones. However, the ElasticSearch date/time formatters at https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-date-format.html still document using the Z format. Which one does ES actually use?

Furthermore, when migrating indices from 6.8.2 to 7.0, will the date fields in the documents also be updated to store the dates using XXX instead of Z?

Thanks!

[lmariselvam (Mariselvam)]

@ben-charles

[epoch_millis||date_time||date_time_no_millis]

@ben-charles, Based on the error reported, you could try either one of these formats:

1. epoch_millis
2. date_time (yyyy-MM-dd'T'HH:mm:ss.SSSZZ). Example: 2020-02-16T09:44:02.342-0400
3. date_time_no_millis (yyyy-MM-dd'T'HH:mm:ssZZ). Example: 2020-02-16T09:44:02-0400

[ben-charles]

@lmariselvam These are the formats I am trying to use. In the error message I posted, it shows that ES is failing to parse this datetime using the same 3 formats you listed.

The error message is:
ElasticsearchParseException[failed to parse date field [1970-01-01T00:00:00.003Z] with format [epoch_millis||date_time||date_time_no_millis]: [Text '1970-01-01T00:00:00.003Z' could not be parsed: Conflict found: Fields resolved to two different dates: -0001-01-01 1970-01-01]]; nested: DateTimeParseException[Text '1970-01-01T00:00:00.003Z' could not be parsed: Conflict found: Fields resolved to two different dates: -0001-01-01 1970-01-01]; nested: DateTimeException[Conflict found: Fields resolved to two different dates: -0001-01-01 1970-01-01];

I am not sure about the fields being resolved to two different dates. Any ideas?

[lmariselvam (Mariselvam)]

@ben-charles It looks the value you're trying to index is 1970-01-01T00:00:00.003Z, Could you try to put the appropriate timezone value in the "Z" part such as 1970-01-01T00:00:00.003-0400 and index the record?