Joda time vs Java Time in 6.8

[stefws (Steffen Winther Sørensen)]

got a working 6.8 cluster ingesting data into indicies with templates that defines some date fields like this:

  "os_installed": {
    "type": "date",
    "format": "MM/DD/YYYY||MM\/DD\/YYYY||MM\\/DD\\/YYYY||strict_year_month_day||strict_date_optional_time||epoch_millis"
  },

As prep for upgrading to 7.x I'm trying to alter templates like this:

  "os_installed": {
    "type": "date",
    "format": "8MM/DD/yyyy||MM\/DD\/yyyy||MM\\/DD\\/yyyy||strict_year_month_day||strict_date_optional_time||epoch_millis"
  },

but only gets a fraction of doc indexed giving dates like this in the index:

"os_installed": "01\\/03\\/2018"

while majority of docs fails to get indexed generating error like this:

[2019-10-15T10:12:48,589][DEBUG][o.e.a.b.TransportShardBulkAction] [d1r1n1] [tanium_basic_inventory-2019.10.15][2] failed to execute bulk item (index) index {[tanium_basic_inventory-2019.10.15][_doc][Zxx6zm0BLyt5VBipDnsa], source[{...redacted...,"os_installed":"03\\/26\\/2014",...redacted..."@timestamp":"2019-10-15T08:12:32.92+00:00"}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse field [os_installed] of type [date] in document with id 'Zxx6zm0BLyt5VBipDnsa'
...
Caused by: java.lang.IllegalArgumentException: failed to parse date field [03\/26\/2014] with format [8MM/DD/yyyy||MM/DD/yyyy||MM\/DD\/yyyy||strict_year_month_day||strict_date_optional_time||epoch_millis]
        at org.elasticsearch.common.time.JavaDateFormatter.parse(JavaDateFormatter.java:116) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.common.time.DateFormatter.parseMillis(DateFormatter.java:54) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.index.mapper.DateFieldMapper$DateFieldType.parse(DateFieldMapper.java:246) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.index.mapper.DateFieldMapper.parseCreateField(DateFieldMapper.java:454) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:297) ~[elasticsearch-6.8.1.jar:6.8.1]

I'm also confused about howto define Java data format specifications in my templates and if such format will continue to work in ES 7.x directly or needs changing in 7.x and if to what 'y' -> 'u' and '8' -> ''?

Tried also this with similar bad ratio between indexed/non-indexed docs:

Caused by: java.lang.IllegalArgumentException: failed to parse date field [09\/02\/2016] with format [8MM/DD/uuuu||MM/DD/uuuu||MM\/DD\/uuuu||strict_year_month_day||strict_date_optional_time||epoch_millis]

TIA for any hints!

[stefws (Steffen Winther Sørensen)]

When trying this limited format specification

"os_installed": {
    "type": "date",
    "format": "8MM\\/DD\\/yyyy"
  },

I'm getting below different error during ingestion:

{"type":"illegal_argument_exception","reason":"failed to parse date field [03\\/14\\/2017] with format [8MM\\/DD\\/yyyy]","caused_by":
  {"type":"date_time_parse_exception","reason":"date_time_parse_exception: Text '03\\/14\\/2017' could not be parsed: Conflict found: Field MonthOfYear 1 differs from MonthOfYear 3 derived from 2017-01-14","caused_by":
    {"type":"date_time_exception","reason":"date_time_exception: Conflict found: Field MonthOfYear 1 differs from MonthOfYear 3 derived from 2017-01-14"
    }
  }
}

Why?

[spinscale (Alexander Reelsen)]

Using DD means you want to extract the current day of the year (sth between 1 and 366) and not the current day of the month (which would have been dd). The day of the year overlaps with specifying the month, as both set the month of the year and this is why this error occurs.

for the first example, can you provide a couple of formats for testing to dig into this?

[stefws (Steffen Winther Sørensen)]

Oh right I see from here:

Letter  Date or Time Component  Presentation    Examples
G       Era designator          Text                AD
y       Year                    Year                1996; 96
M       Month in year           Month               July; Jul; 07
w       Week in year            Number              27
W       Week in month           Number              2
D       Day in year             Number              189
d       Day in month            Number              10
F       Day of week in month    Number              2
E       Day in week             Text                Tuesday; Tue
a       Am/pm marker            Text                PM
H       Hour in day (0-23)      Number              0
k       Hour in day (1-24)      Number              24
K       Hour in am/pm (0-11)    Number              0
h       Hour in am/pm (1-12)    Number              12
m       Minute in hour          Number              30
s       Second in minute        Number              55
S       Millisecond             Number              978
z       Time zone               General time zone   Pacific Standard Time; PST; GMT-08:00
Z       Time zone               RFC 822 time zone   -0800 

Migration doc should also be more specific whether or not to prefix with 8 on every format specifier or just once for the hole format value string... Edit: It seems only to be once for the hole format value string and not for every specifier item and using 'u' instead of 'y' makes Migration Assistant happy like so:

"os_installed": {
    "type": "date",
    "format": "8MM/DD/uuuu||MM\/DD\/uuuu||MM\\/DD\\/uuuu||strict_year_month_day||strict_date_optional_time||epoch_millis"
  },