I'm working with ES 6.8.1 (and ditto for the rest of the Elastic Stack) that's using an index template that Logstash sends to for indexing. There are some date fields in the index template where I'm seeing the declaration for the date type fields not honoring (or I'm misunderstanding or just doing it plain wrong) the defined date formats. The indices created are done so dynamically by plucking off the year and month (yyyy-MM) from a date field and then appended to the index name, which the index template matches against.
I do have this index template asserting
"dynamic":"strict" in the mapping properties, but I feel like this shouldn't have any effect on having multiple date formats on the date fields. And I haven't stumbled on any issues or literature noting that this might cause problems beyond rejecting events from indexing if not all the fields presented are known to the template.
The date fields themselves all look like so:
"format": "yyyy-MM-dd HH:mm:ss.SSSSSSSS||yyyy-MM-dd HH:mm:ss"
In my Logstash logs I see all events with date fields for the matching index template asserting the date field value is too short. I've tested this by dropping the existing index and letting the workflow recreate it (it's based on month) again to ensure I didn't gloss over any index mapping changes that might've occurred. I also did a quick query on the index mapping schema itself and saw the date fields were as I note above. I get the same output from Logstash about being unable to index after dropping and letting the index recreate.
So this makes me consider two things:
Elasticsearch is doing some sort of short circuit evaluation of the date formats it encounters when attempting to index. If it doesn't match the leftmost format, it naively thinks it's not going to match anything defined afterward (very unlikely).
I'm not defining multiple possible date formats correctly. As far as the documentation goes, I haven't done anything obviously wrong or different compared to both 7.x current and 6.8. Using double pipes to designate an OR condition is the right syntax. I'm not sure why it seems to be short circuit evaluating things.
When I flip the expected date formats around from above, things index properly. However, I'm trying to future proof the index template to anticipate microseconds.
Anything jump out to you all on what's up here? I do believe the assertion for microseconds is valid since the Logstash errors are stating "too short" strongly suggesting it's actually expecting the microseconds to be there. Thanks in advance to any suggestions that crop up!