# Error during elasticsearch upgrade 6.8 to 7.4.2 unassigned shards IllegalArgumentException failed to parse date field

**URL:** <https://discuss.elastic.co/t/error-during-elasticsearch-upgrade-6-8-to-7-4-2-unassigned-shards-illegalargumentexception-failed-to-parse-date-field/209452>\
**Category:** Elasticsearch\
**Created:** [November 26, 2019, 9:06am UTC](https://discuss.elastic.co/t/error-during-elasticsearch-upgrade-6-8-to-7-4-2-unassigned-shards-illegalargumentexception-failed-to-parse-date-field/209452 "2019-11-26T09:06:57Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Petr.Simik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/petr.simik/32/38082_2.png) [@Petr.Simik](https://discuss.elastic.co/u/Petr.Simik)\
**Post date:** [November 26, 2019, 9:06am UTC](https://discuss.elastic.co/t/error-during-elasticsearch-upgrade-6-8-to-7-4-2-unassigned-shards-illegalargumentexception-failed-to-parse-date-field/209452/1 "2019-11-26T09:06:57Z")

</div>

ERROR: (IllegalArgumentException[failed to parse date field [1574755226.3741724] with format [strict\_date\_optional\_time||epoch\_millis]])

During upgrade after I upgraded one node and reenabled shard allocation

> PUT \_cluster/settings  
> {  
> "persistent": {  
> "cluster.routing.allocation.enable": "all"  
> }  
> }

I received unassigned shards (yellow state of cluster)  
The log files and GET \_cluster/allocation/explain show following error.

> ```
> {
> "node_id" : "l1Hxb5gnRF224OYPNqM3PA",
> "node_name" : "node03",
> "transport_address" : "10.10.10.13:9300",
> "node_attributes" : {
> "ml.machine_memory" : "67387482112",
> "ml.max_open_jobs" : "20",
> "xpack.installed" : "true"
> },
> "node_decision" : "no",
> "deciders" : [
> {
> "decider" : "max_retry",
> "decision" : "NO",
> "explanation" : "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2019-11-26T08:03:56.934Z], failed_attempts[5], delayed=false, details[failed shard on node [xIiNk12YR6K8ZMfA_Af7JA]: failed to perform indices:data/write/bulk[s] on replica [index_aggreg_00865][2], node[xIiNk12YR6K8ZMfA_Af7JA], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=pESCi8GATOqZ9ZrsMdWVOQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2019-11-26T08:03:47.268Z], failed_attempts[4], delayed=false, details[failed shard on node [xIiNk12YR6K8ZMfA_Af7JA]: failed to perform indices:data/write/bulk[s] on replica [index_aggreg_00865][2], node[xIiNk12YR6K8ZMfA_Af7JA], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=-FIfJRSCSxCq-uW15YifpQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2019-11-26T08:03:40.984Z], failed_attempts[3], delayed=false, details[failed shard on node [xIiNk12YR6K8ZMfA_Af7JA]: failed to perform indices:data/write/bulk[s] on replica [index_aggreg_00865][2], node[xIiNk12YR6K8ZMfA_Af7JA], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=hV1UZo0FRpSWx6VzfJYbvw], unassigned_info[[reason=ALLOCATION_FAILED], at[2019-11-26T08:03:25.956Z], failed_attempts[2], delayed=false, details[failed shard on node [xIiNk12YR6K8ZMfA_Af7JA]: failed to perform indices:data/write/bulk[s] on replica [index_aggreg_00865][2], node[xIiNk12YR6K8ZMfA_Af7JA], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=CHODudw1T5KCfm0UkDNXDA], unassigned_info[[reason=ALLOCATION_FAILED], at[2019-11-26T08:03:20.546Z], failed_attempts[1], delayed=false, details[failed shard on node [xIiNk12YR6K8ZMfA_Af7JA]: failed to perform indices:data/write/bulk[s] on replica [index_aggreg_00865][2], node[xIiNk12YR6K8ZMfA_Af7JA], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=CXQETlhcQ265Lvpd_LK0mg], unassigned_info[[reason=REPLICA_ADDED], at[2019-11-26T08:03:14.282Z], delayed=false, allocation_status[no_attempt]], failure RemoteTransportException[[tela04-prahkz][10.10.10.14:9300][indices:data/write/bulk[s][r]]]; nested: MapperParsingException[failed to parse field [slice] of type [date] in document with id 'LUy8pm4BW1hCj0_ie8ZG'. Preview of field's value: '1.5747552263741724E9']; nested: IllegalArgumentException[failed to parse date field [1574755226.3741724] with format [strict_date_optional_time||epoch_millis]]; nested: NotSerializableExceptionWrapper[date_time_parse_exception: Failed to parse with all enclosed parsers]; ], allocation_status[no_attempt]], failure RemoteTransportException[[tela04-prahkz][10.10.10.14:9300][indices:data/write/bulk[s][r]]]; nested: MapperParsingException[failed to parse field [slice] of type [date] in document with id 'r0y8pm4BW1hCj0_ikM1s'. Preview of field's value: '1.5747552263741724E9']; nested: IllegalArgumentException[failed to parse date field [1574755226.3741724] with format [strict_date_optional_time||epoch_millis]]; nested: NotSerializableExceptionWrapper[date_time_parse_exception: Failed to parse with all enclosed parsers]; ], allocation_status[no_attempt]], failure RemoteTransportException[[tela04-prahkz][10.10.10.14:9300][indices:data/write/bulk[s][r]]]; nested: MapperParsingException[failed to parse field [slice] of type [date] in document with id 'w0y8pm4BW1hCj0_iy98Y'. Preview of field's value: '1.5747552263741724E9']; nested: IllegalArgumentException[failed to parse date field [1574755226.3741724] with format [strict_date_optional_time||epoch_millis]]; nested: NotSerializableExceptionWrapper[date_time_parse_exception: Failed to parse with all enclosed parsers]; ], allocation_status[no_attempt]], failure RemoteTransportException[[tela04-prahkz][10.10.10.14:9300][indices:data/write/bulk[s][r]]]; nested: MapperParsingException[failed to parse field [slice] of type [date] in document with id 'UR-8pm4Bfu7W77kn468S'. Preview of field's value: '1.5747552263741724E9']; nested: IllegalArgumentException[failed to parse date field [1574755226.3741724] with format [strict_date_optional_time||epoch_millis]]; nested: NotSerializableExceptionWrapper[date_time_parse_exception: Failed to parse with all enclosed parsers]; ], allocation_status[no_attempt]], failure RemoteTransportException[[tela04-prahkz][10.10.10.14:9300][indices:data/write/bulk[s][r]]]; nested: MapperParsingException[failed to parse field [slice] of type [date] in document with id 'Sii9pm4B3G181yzJCV5e'. Preview of field's value: '1.5747552263741724E9']; nested: IllegalArgumentException[failed to parse date field [1574755226.3741724] with format [strict_date_optional_time||epoch_millis]]; nested: NotSerializableExceptionWrapper[date_time_parse_exception: Failed to parse with all enclosed parsers]; ], allocation_status[no_attempt]]]"
> },
> {
> "decider" : "throttling",
> "decision" : "THROTTLE",
> "explanation" : "reached the limit of outgoing shard recoveries [2] on the node [g2bPDi1XRcSuxYY5XAQwDQ] which holds the primary, cluster setting [cluster.routing.allocation.node_concurrent_outgoing_recoveries=2] (can also be set via [cluster.routing.allocation.node_concurrent_recoveries])"
> }
> ]
> },
> 
> ```

no idea why  
I tried to create on upgraded node test index with same mapping and put the same value it is complaining and it succesfully inserted it.

---

<div class="post-metadata">

**Author:** ![Petr.Simik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/petr.simik/32/38082_2.png) [@Petr.Simik](https://discuss.elastic.co/u/Petr.Simik)\
**Post date:** [November 26, 2019, 2:42pm UTC](https://discuss.elastic.co/t/error-during-elasticsearch-upgrade-6-8-to-7-4-2-unassigned-shards-illegalargumentexception-failed-to-parse-date-field/209452/2 "2019-11-26T14:42:43Z")

</div>

I was able to reproduce the error:

I created the index using the template  
and I create document within this index with problematic field slice

> PUT product\_aggreg\_001
> 
> GET \_cat/shards/product\_aggreg\_001
> 
> product\_aggreg\_001 4 r STARTED anode04-prahkz  
> product\_aggreg\_001 4 p STARTED node-physical5  
> product\_aggreg\_001 3 p STARTED node-physical5  
> product\_aggreg\_001 3 r STARTED anode01-prahkz  
> product\_aggreg\_001 1 r STARTED anode02-prahkz  
> product\_aggreg\_001 1 p STARTED node-physical5  
> product\_aggreg\_001 2 r STARTED anode04-prahkz  
> product\_aggreg\_001 2 p STARTED node-physical5  
> product\_aggreg\_001 0 r STARTED anode04-prahkz  
> product\_aggreg\_001 0 p STARTED node-physical5

#Error

> ```
> PUT product_aggreg_001/_doc/1
> { "slice": 1.574733608581309E9 }
> PUT product_aggreg_001/_doc/21
> { "slice": 1574744407.1290631 }
> 
> ```

it returns error when inserting record on upgraded node (xIiNk12YR6K8ZMfA\_Af7JA = upgraded node)

> #! Deprecation: Use of scientific notation in epoch time formats is deprecated and will not be supported in the next major version of Elasticsearch.  
> {  
> "\_index" : "product\_aggreg\_001",  
> "\_type" : "\_doc",  
> "\_id" : "1",  
> "\_version" : 1,  
> "result" : "created",  
> "\_shards" : {  
> "total" : 2,  
> "successful" : 1,  
> "failed" : 1,  
> "failures" : [  
> {  
> "\_index" : "product\_aggreg\_001",  
> "\_shard" : 3,  
> "\_node" : "xIiNk12YR6K8ZMfA\_Af7JA",  
> "reason" : {  
> "type" : "mapper\_parsing\_exception",  
> "reason" : "failed to parse field [slice] of type [date] in document with id '1'. Preview of field's value: '1.574733608581309E9'",  
> "caused\_by" : {  
> "type" : "illegal\_argument\_exception",  
> "reason" : "failed to parse date field [1.574733608581309E9] with format [strict\_date\_optional\_time||epoch\_millis]",  
> "caused\_by" : {  
> "type" : "date\_time\_parse\_exception",  
> "reason" : "date\_time\_parse\_exception: Failed to parse with all enclosed parsers"  
> }  
> }  
> },  
> "status" : "BAD\_REQUEST",  
> "primary" : false  
> }  
> ]  
> },  
> "\_seq\_no" : 0,  
> "\_primary\_term" : 1  
> }

template:

> #! Deprecation: [types removal] The parameter include\_type\_name should be explicitly specified in get template requests to prepare for 7.0. In 7.0 include\_type\_name will default to 'false', which means responses will omit the type name in mapping definitions.  
> {  
> "bs\_product\_aggreg" : {  
> "order" : 0,  
> "version" : 0,  
> "index\_patterns" : [  
> "product\_aggreg\_\*"  
> ],  
> "settings" : {  
> "index" : {  
> "mapping" : {  
> "total\_fields" : {  
> "limit" : "100000"  
> }  
> }  
> }  
> },  
> "mappings" : {  
> "doc" : {  
> "properties" : {  
> "broadcast\_type" : {  
> "type" : "text",  
> "fields" : {  
> "keyword" : {  
> "type" : "keyword",  
> "ignore\_above" : 256  
> }  
> }  
> },  
> "channel\_id" : {  
> "type" : "text",  
> "fields" : {  
> "keyword" : {  
> "type" : "keyword",  
> "ignore\_above" : 256  
> }  
> }  
> },  
> "device\_detail" : {  
> "type" : "text"  
> },  
> "device\_id" : {  
> "type" : "text",  
> "fields" : {  
> "keyword" : {  
> "type" : "keyword",  
> "ignore\_above" : 256  
> }  
> }  
> },  
> "device\_mode" : {  
> "type" : "text",  
> "fields" : {  
> "keyword" : {  
> "type" : "keyword",  
> "ignore\_above" : 256  
> }  
> }  
> },  
> "device\_type" : {  
> "type" : "text",  
> "fields" : {  
> "keyword" : {  
> "type" : "keyword",  
> "ignore\_above" : 256  
> }  
> }  
> },  
> "edr\_id" : {  
> "type" : "text",  
> "fields" : {  
> "keyword" : {  
> "type" : "keyword",  
> "ignore\_above" : 256  
> }  
> }  
> },  
> "entity\_id" : {  
> "type" : "long"  
> },  
> "epg\_id" : {  
> "type" : "long"  
> },  
> "session\_start" : {  
> "type" : "date"  
> },  
> "event\_start" : {  
> "type" : "date"  
> },  
> "event\_end" : {  
> "type" : "date"  
> },  
> "event\_type" : {  
> "type" : "text",  
> "fields" : {  
> "keyword" : {  
> "type" : "keyword",  
> "ignore\_above" : 256  
> }  
> }  
> },  
> "ip\_address" : {  
> "type" : "text",  
> "fields" : {  
> "keyword" : {  
> "type" : "keyword",  
> "ignore\_above" : 256  
> }  
> }  
> },  
> "isp\_id" : {  
> "type" : "long"  
> },  
> "event\_source" : {  
> "type" : "text",  
> "fields" : {  
> "keyword" : {  
> "type" : "keyword",  
> "ignore\_above" : 256  
> }  
> }  
> },  
> "subscription\_code" : {  
> "type" : "text",  
> "fields" : {  
> "keyword" : {  
> "type" : "keyword",  
> "ignore\_above" : 256  
> }  
> }  
> },  
> "slice" : {  
> "type" : "date"  
> }  
> }  
> }  
> },  
> "aliases" : { }  
> }  
> }

---

<div class="post-metadata">

**Author:** ![Petr.Simik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/petr.simik/32/38082_2.png) [@Petr.Simik](https://discuss.elastic.co/u/Petr.Simik)\
**Post date:** [November 27, 2019, 7:59am UTC](https://discuss.elastic.co/t/error-during-elasticsearch-upgrade-6-8-to-7-4-2-unassigned-shards-illegalargumentexception-failed-to-parse-date-field/209452/3 "2019-11-27T07:59:42Z")

</div>

I found the resolution

version 6.8 has accepted date in format unixtime  
1574755226.3741724  
or  
1.5747444071290631E9

but it does not work in v7.x and it end up with error

> failed to parse date field [1.574733608581309E9] with format [strict\_date\_optional\_time||epoch\_millis]

so i changed the type from date to long and reindexed this index.

the problem was caused on index having master on 6.8 and replica upgraded on 7.4 during upgrade 6.8 master obtained new data and those data were not replicated to replica and ended up with this error.

The users did not insist on date format of this field this is why I survived with change to long in index template.  
But there is a resolution in 7.4 which I havent studied in depth..

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 25, 2019, 7:59am UTC](https://discuss.elastic.co/t/error-during-elasticsearch-upgrade-6-8-to-7-4-2-unassigned-shards-illegalargumentexception-failed-to-parse-date-field/209452/4 "2019-12-25T07:59:56Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.