Indexing problem when missing milliseconds for strict_date_time field

Elastic Stack Elasticsearch
1

We have a problem with Elasticsearch 2.3.2.

When indexing a document that contains a Java Instant date with zero milliseconds (eg. Instant.now().truncatedTo(ChronoUnit.SECONDS) ) using the DateTimeFormatter.ISO_INSTANT and when the field is mapped as:
"creationTime": {
"format": "strict_date_time||epoch_millis",
"type": "date"
}
we get the following exception from Elasticsearch:

Exception in thread "main" java.io.IOException: MapperParsingException[failed to parse [creationTime]]; nested: IllegalArgumentException[Invalid format: "2016-05-25T09:37:47Z" is malformed at "Z"];
at com.swift.amw.qgate.authentication.providers.es.ESUserRegistry.createUser(ESUserRegistry.java:96)
at com.swift.amw.qgate.authentication.providers.es.ESUserRegistry.addUser(ESUserRegistry.java:113)
at com.swift.amw.qgate.authentication.providers.es.ESUserRegistry.addUser(ESUserRegistry.java:130)
at com.swift.amw.qgate.authentication.providers.es.ESUserRegistryTest.main(ESUserRegistryTest.java:244)
Caused by: MapperParsingException[failed to parse [creationTime]]; nested: IllegalArgumentException[Invalid format: "2016-05-25T09:37:47Z" is malformed at "Z"];
at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:329)
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:309)
at org.elasticsearch.index.mapper.DocumentParser.parseValue(DocumentParser.java:436)
at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:262)
at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:122)
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:309)
at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:580)
at org.elasticsearch.index.shard.IndexShard.prepareIndexOnPrimary(IndexShard.java:559)
at org.elasticsearch.action.index.TransportIndexAction.prepareIndexOperationOnPrimary(TransportIndexAction.java:212)
at org.elasticsearch.action.index.TransportIndexAction.executeIndexRequestOnPrimary(TransportIndexAction.java:224)
at org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:158)
at org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:66)
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.doRun(TransportReplicationAction.java:639)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:279)
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:271)
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75)
at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Invalid format: "2016-05-25T09:37:47Z" is malformed at "Z"
at org.joda.time.format.DateTimeParserBucket.doParseMillis(DateTimeParserBucket.java:187)
at org.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:780)
at org.elasticsearch.index.mapper.core.DateFieldMapper$DateFieldType.parseStringValue(DateFieldMapper.java:362)
at org.elasticsearch.index.mapper.core.DateFieldMapper.innerParseCreateField(DateFieldMapper.java:528)
at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:241)
at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:321)
... 21 more

The DateTimeFormatter.ISO_INSTANT formatter drops the milliseconds from the generated string when they are zero. When having a value for milliseconds that is larger then zero, all is working fine.
When changing the mapping to
"creationTime": {
"format": "strict_date_optional_time||epoch_millis",
"type": "date"
}
all is working correctly.

Is this a correct behaviour? I want my field to always have a time part but allow for 0 milliseconds.

2

That appears to be correct behaviour, yes.

3

I believe the DateFieldMapper should invoke the org.joda.time formatter to parse the string more lenient. Or even better, migrate to java.time :wink:

I have put in place a custom jackson Instant serializer that allows write out milliseconds, even when zero. Now, I can put back the strict_date_time format for my date fields.

4

Feel free to raise this on GH, we're always after ways to improve things! :slight_smile: