I have 6 node data node cluster. I started an index restore with Replica 1. But after some time the status of the index is yellow then when I checked the health status it returned many shareds are unassigned.
From the explain allocation, I got this message.
"explanation" : "the node is above the low watermark cluster setting [cluster.routing.allocation.disk.watermark.low=85%], using more disk space than the maximum allowed [85.0%], actual free: [14.148040008774746%]"
The disk is reached 85% on a few nodes. Other nodes are reached 70%. But somehow it didn't work. So I increase the watermark to 90% and executed the /_cluster/reroute?retry_failed=true. It was fine for sometimes. Then I got the below error message.
{
"node_id" : "eyfd61iIROKRZWQZNNJNVw",
"node_name" : "node-6",
"transport_address" : "10.10.10.10:9300",
"node_decision" : "no",
"deciders" : [
{
"decider" : "max_retry",
"decision" : "NO",
"explanation" : "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2020-06-19T19:06:20.446Z], failed_attempts[5], delayed=false, details[failed to create index, failure IllegalArgumentException[Unknown char_filter type [icu_normalizer] for [icu_normalizer_casesensitive]]], allocation_status[no_attempt]]]"
}
]
},
I also tried the reroute. but got the below response.
{
"error": {
"root_cause": [
{
"type": "remote_transport_exception",
"reason": "[node-2][10.10.10.6:9300][cluster:admin/reroute]"
}
],
"type": "illegal_argument_exception",
"reason": "[allocate_replica] allocation of [index-01][1] on node {prod-onclusive-es-d-us-es4-b-08-gce}{wcF3Ob3ATKu5jB3_ur2k8w}{UDpuIGc_SUGtj4K7BOaIOw}{10.210.80.12}{10.210.80.12:9300} is not allowed, reason: [NO(shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2020-06-19T19:50:53.042Z], failed_attempts[6], delayed=false, details[failed to create index, failure IllegalArgumentException[Unknown char_filter type [icu_normalizer] for [icu_normalizer_casesensitive]]], allocation_status[no_attempt]]])][YES(primary shard for this replica is already active)][YES(explicitly ignoring any disabling of allocation due to manual allocation commands via the reroute API)][YES(target node version [5.6.16] is the same or newer than source node version [5.6.16])][YES(the shard is not being snapshotted)][YES(ignored as shard is not being recovered from a snapshot)][YES(node passes include/exclude/require filters)][YES(the shard does not exist on the same node)][YES(enough disk for shard on node, free: [457.6gb], shard size: [0b], free after allocating shard: [457.6gb])][YES(below shard recovery limit of outgoing: [0 < 2] incoming: [0 < 2])][YES(total shard limits are disabled: [index: -1, cluster: -1] <= 0)][YES(allocation awareness is not enabled, set cluster setting [cluster.routing.allocation.awareness.attributes] to enable it)]"
},
"status": 400
}
There are some useful links I tried(Shard allocation says max retry but fails to allocate on retry_failed=true), but no luck.
Can someone help me with how to solve this? And 1st it is a disk space issue, but next time nothing, then how it went to unassigned?
ES version 5.6
Total data size: 3TB
Master nodes: 3
Data node: 6
Heap on data node: 20GB