got a working 6.8 cluster ingesting data into indicies with templates that defines some date fields like this:
"os_installed": {
"type": "date",
"format": "MM/DD/YYYY||MM\/DD\/YYYY||MM\\/DD\\/YYYY||strict_year_month_day||strict_date_optional_time||epoch_millis"
},
As prep for upgrading to 7.x I'm trying to alter templates like this:
"os_installed": {
"type": "date",
"format": "8MM/DD/yyyy||MM\/DD\/yyyy||MM\\/DD\\/yyyy||strict_year_month_day||strict_date_optional_time||epoch_millis"
},
but only gets a fraction of doc indexed giving dates like this in the index:
"os_installed": "01\\/03\\/2018"
while majority of docs fails to get indexed generating error like this:
[2019-10-15T10:12:48,589][DEBUG][o.e.a.b.TransportShardBulkAction] [d1r1n1] [tanium_basic_inventory-2019.10.15][2] failed to execute bulk item (index) index {[tanium_basic_inventory-2019.10.15][_doc][Zxx6zm0BLyt5VBipDnsa], source[{...redacted...,"os_installed":"03\\/26\\/2014",...redacted..."@timestamp":"2019-10-15T08:12:32.92+00:00"}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse field [os_installed] of type [date] in document with id 'Zxx6zm0BLyt5VBipDnsa'
...
Caused by: java.lang.IllegalArgumentException: failed to parse date field [03\/26\/2014] with format [8MM/DD/yyyy||MM/DD/yyyy||MM\/DD\/yyyy||strict_year_month_day||strict_date_optional_time||epoch_millis]
at org.elasticsearch.common.time.JavaDateFormatter.parse(JavaDateFormatter.java:116) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.common.time.DateFormatter.parseMillis(DateFormatter.java:54) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.index.mapper.DateFieldMapper$DateFieldType.parse(DateFieldMapper.java:246) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.index.mapper.DateFieldMapper.parseCreateField(DateFieldMapper.java:454) ~[elasticsearch-6.8.1.jar:6.8.1]
at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:297) ~[elasticsearch-6.8.1.jar:6.8.1]
I'm also confused about howto define Java data format specifications in my templates and if such format will continue to work in ES 7.x directly or needs changing in 7.x and if to what 'y' -> 'u' and '8' -> ''?
Tried also this with similar bad ratio between indexed/non-indexed docs:
Caused by: java.lang.IllegalArgumentException: failed to parse date field [09\/02\/2016] with format [8MM/DD/uuuu||MM/DD/uuuu||MM\/DD\/uuuu||strict_year_month_day||strict_date_optional_time||epoch_millis]
TIA for any hints!