Cluster.routing.allocation.awareness cause shards unassigned

[jayqie]

I split one elasticsearch cluster into two groups for different scenario usage. One is for high-frequency calls, and the other is normal. The two groups are isolated to prevent the normal group from being affected. I split the cluster using following steps:

1. Create a elasticsearch cluster with four datanodes, two are high-frequency calls, and two are normal.
   data-1: node.attr.cluster: highfreq
   data-2: node.attr.cluster: normalfreq
   data-3: node.attr.cluster: highfreq
   data-4: node.attr.cluster: normalfreq

2. Create a new index, routing to two groups
   PUT /ztest
   {
   "settings": {
   "number_of_shards": 3,
   "number_of_replicas": 1,
   "index.routing.allocation.include.cluster": "highfreq,normalfreq"
   }
   }

3. Make sure that each group has a complete copy of shards.
   PUT /_cluster/settings
   {
   "persistent" : {
   "cluster.routing.allocation.awareness.attributes" : "cluster"
   }
   }

4. Specify the group when querying
   GET /ztest/_search?preference=cluster:highfreq
   GET /ztest/_search?preference=cluster:normal

It seems to work properly:

5. But sometimes I want to allocate the index only in normal group(this often occurs in some scenarios):

PUT /ztest
{
"settings": {
"number_of_shards": 3,
"number_of_replicas": 1,
"index.routing.allocation.include.cluster": "normalfreq"
}
}

Then only primary shards can be allocated, and the replication shards are unassigned, although there is still a spare resource of one data-node: