Error during elasticsearch upgrade 6.8 to 7.4.2 unassigned shards IllegalArgumentException failed to parse date field

Elastic Stack Elasticsearch
1

ERROR: (IllegalArgumentException[failed to parse date field [1574755226.3741724] with format [strict_date_optional_time||epoch_millis]])

During upgrade after I upgraded one node and reenabled shard allocation

PUT _cluster/settings
{
"persistent": {
"cluster.routing.allocation.enable": "all"
}
}

I received unassigned shards (yellow state of cluster)
The log files and GET _cluster/allocation/explain show following error.

{
  "node_id" : "l1Hxb5gnRF224OYPNqM3PA",
  "node_name" : "node03",
  "transport_address" : "10.10.10.13:9300",
  "node_attributes" : {
    "ml.machine_memory" : "67387482112",
    "ml.max_open_jobs" : "20",
    "xpack.installed" : "true"
  },
  "node_decision" : "no",
  "deciders" : [
    {
      "decider" : "max_retry",
      "decision" : "NO",
      "explanation" : "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2019-11-26T08:03:56.934Z], failed_attempts[5], delayed=false, details[failed shard on node [xIiNk12YR6K8ZMfA_Af7JA]: failed to perform indices:data/write/bulk[s] on replica [index_aggreg_00865][2], node[xIiNk12YR6K8ZMfA_Af7JA], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=pESCi8GATOqZ9ZrsMdWVOQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2019-11-26T08:03:47.268Z], failed_attempts[4], delayed=false, details[failed shard on node [xIiNk12YR6K8ZMfA_Af7JA]: failed to perform indices:data/write/bulk[s] on replica [index_aggreg_00865][2], node[xIiNk12YR6K8ZMfA_Af7JA], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=-FIfJRSCSxCq-uW15YifpQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2019-11-26T08:03:40.984Z], failed_attempts[3], delayed=false, details[failed shard on node [xIiNk12YR6K8ZMfA_Af7JA]: failed to perform indices:data/write/bulk[s] on replica [index_aggreg_00865][2], node[xIiNk12YR6K8ZMfA_Af7JA], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=hV1UZo0FRpSWx6VzfJYbvw], unassigned_info[[reason=ALLOCATION_FAILED], at[2019-11-26T08:03:25.956Z], failed_attempts[2], delayed=false, details[failed shard on node [xIiNk12YR6K8ZMfA_Af7JA]: failed to perform indices:data/write/bulk[s] on replica [index_aggreg_00865][2], node[xIiNk12YR6K8ZMfA_Af7JA], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=CHODudw1T5KCfm0UkDNXDA], unassigned_info[[reason=ALLOCATION_FAILED], at[2019-11-26T08:03:20.546Z], failed_attempts[1], delayed=false, details[failed shard on node [xIiNk12YR6K8ZMfA_Af7JA]: failed to perform indices:data/write/bulk[s] on replica [index_aggreg_00865][2], node[xIiNk12YR6K8ZMfA_Af7JA], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=CXQETlhcQ265Lvpd_LK0mg], unassigned_info[[reason=REPLICA_ADDED], at[2019-11-26T08:03:14.282Z], delayed=false, allocation_status[no_attempt]], failure RemoteTransportException[[tela04-prahkz][10.10.10.14:9300][indices:data/write/bulk[s][r]]]; nested: MapperParsingException[failed to parse field [slice] of type [date] in document with id 'LUy8pm4BW1hCj0_ie8ZG'. Preview of field's value: '1.5747552263741724E9']; nested: IllegalArgumentException[failed to parse date field [1574755226.3741724] with format [strict_date_optional_time||epoch_millis]]; nested: NotSerializableExceptionWrapper[date_time_parse_exception: Failed to parse with all enclosed parsers]; ], allocation_status[no_attempt]], failure RemoteTransportException[[tela04-prahkz][10.10.10.14:9300][indices:data/write/bulk[s][r]]]; nested: MapperParsingException[failed to parse field [slice] of type [date] in document with id 'r0y8pm4BW1hCj0_ikM1s'. Preview of field's value: '1.5747552263741724E9']; nested: IllegalArgumentException[failed to parse date field [1574755226.3741724] with format [strict_date_optional_time||epoch_millis]]; nested: NotSerializableExceptionWrapper[date_time_parse_exception: Failed to parse with all enclosed parsers]; ], allocation_status[no_attempt]], failure RemoteTransportException[[tela04-prahkz][10.10.10.14:9300][indices:data/write/bulk[s][r]]]; nested: MapperParsingException[failed to parse field [slice] of type [date] in document with id 'w0y8pm4BW1hCj0_iy98Y'. Preview of field's value: '1.5747552263741724E9']; nested: IllegalArgumentException[failed to parse date field [1574755226.3741724] with format [strict_date_optional_time||epoch_millis]]; nested: NotSerializableExceptionWrapper[date_time_parse_exception: Failed to parse with all enclosed parsers]; ], allocation_status[no_attempt]], failure RemoteTransportException[[tela04-prahkz][10.10.10.14:9300][indices:data/write/bulk[s][r]]]; nested: MapperParsingException[failed to parse field [slice] of type [date] in document with id 'UR-8pm4Bfu7W77kn468S'. Preview of field's value: '1.5747552263741724E9']; nested: IllegalArgumentException[failed to parse date field [1574755226.3741724] with format [strict_date_optional_time||epoch_millis]]; nested: NotSerializableExceptionWrapper[date_time_parse_exception: Failed to parse with all enclosed parsers]; ], allocation_status[no_attempt]], failure RemoteTransportException[[tela04-prahkz][10.10.10.14:9300][indices:data/write/bulk[s][r]]]; nested: MapperParsingException[failed to parse field [slice] of type [date] in document with id 'Sii9pm4B3G181yzJCV5e'. Preview of field's value: '1.5747552263741724E9']; nested: IllegalArgumentException[failed to parse date field [1574755226.3741724] with format [strict_date_optional_time||epoch_millis]]; nested: NotSerializableExceptionWrapper[date_time_parse_exception: Failed to parse with all enclosed parsers]; ], allocation_status[no_attempt]]]"
    },
    {
      "decider" : "throttling",
      "decision" : "THROTTLE",
      "explanation" : "reached the limit of outgoing shard recoveries [2] on the node [g2bPDi1XRcSuxYY5XAQwDQ] which holds the primary, cluster setting [cluster.routing.allocation.node_concurrent_outgoing_recoveries=2] (can also be set via [cluster.routing.allocation.node_concurrent_recoveries])"
    }
  ]
},

no idea why
I tried to create on upgraded node test index with same mapping and put the same value it is complaining and it succesfully inserted it.

2

I was able to reproduce the error:

I created the index using the template
and I create document within this index with problematic field slice

PUT product_aggreg_001

GET _cat/shards/product_aggreg_001

product_aggreg_001 4 r STARTED anode04-prahkz
product_aggreg_001 4 p STARTED node-physical5
product_aggreg_001 3 p STARTED node-physical5
product_aggreg_001 3 r STARTED anode01-prahkz
product_aggreg_001 1 r STARTED anode02-prahkz
product_aggreg_001 1 p STARTED node-physical5
product_aggreg_001 2 r STARTED anode04-prahkz
product_aggreg_001 2 p STARTED node-physical5
product_aggreg_001 0 r STARTED anode04-prahkz
product_aggreg_001 0 p STARTED node-physical5

#Error

PUT product_aggreg_001/_doc/1
{ "slice": 1.574733608581309E9 }
PUT product_aggreg_001/_doc/21
{ "slice": 1574744407.1290631 }

it returns error when inserting record on upgraded node (xIiNk12YR6K8ZMfA_Af7JA = upgraded node)

#! Deprecation: Use of scientific notation in epoch time formats is deprecated and will not be supported in the next major version of Elasticsearch.
{
"_index" : "product_aggreg_001",
"_type" : "_doc",
"_id" : "1",
"_version" : 1,
"result" : "created",
"_shards" : {
"total" : 2,
"successful" : 1,
"failed" : 1,
"failures" : [
{
"_index" : "product_aggreg_001",
"_shard" : 3,
"_node" : "xIiNk12YR6K8ZMfA_Af7JA",
"reason" : {
"type" : "mapper_parsing_exception",
"reason" : "failed to parse field [slice] of type [date] in document with id '1'. Preview of field's value: '1.574733608581309E9'",
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "failed to parse date field [1.574733608581309E9] with format [strict_date_optional_time||epoch_millis]",
"caused_by" : {
"type" : "date_time_parse_exception",
"reason" : "date_time_parse_exception: Failed to parse with all enclosed parsers"
}
}
},
"status" : "BAD_REQUEST",
"primary" : false
}
]
},
"_seq_no" : 0,
"_primary_term" : 1
}

template:

#! Deprecation: [types removal] The parameter include_type_name should be explicitly specified in get template requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', which means responses will omit the type name in mapping definitions.
{
"bs_product_aggreg" : {
"order" : 0,
"version" : 0,
"index_patterns" : [
"product_aggreg_*"
],
"settings" : {
"index" : {
"mapping" : {
"total_fields" : {
"limit" : "100000"
}
}
}
},
"mappings" : {
"doc" : {
"properties" : {
"broadcast_type" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"channel_id" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"device_detail" : {
"type" : "text"
},
"device_id" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"device_mode" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"device_type" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"edr_id" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"entity_id" : {
"type" : "long"
},
"epg_id" : {
"type" : "long"
},
"session_start" : {
"type" : "date"
},
"event_start" : {
"type" : "date"
},
"event_end" : {
"type" : "date"
},
"event_type" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"ip_address" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"isp_id" : {
"type" : "long"
},
"event_source" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"subscription_code" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"slice" : {
"type" : "date"
}
}
}
},
"aliases" : { }
}
}

3

I found the resolution

version 6.8 has accepted date in format unixtime
1574755226.3741724
or
1.5747444071290631E9

but it does not work in v7.x and it end up with error

failed to parse date field [1.574733608581309E9] with format [strict_date_optional_time||epoch_millis]

so i changed the type from date to long and reindexed this index.

the problem was caused on index having master on 6.8 and replica upgraded on 7.4 during upgrade 6.8 master obtained new data and those data were not replicated to replica and ended up with this error.

The users did not insist on date format of this field this is why I survived with change to long in index template.
But there is a resolution in 7.4 which I havent studied in depth..

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.