Forced awareness + replica count

[priteshkp (Pritesh Patel)]

Using ES 6.1.0.

I'm trying to understand what might be happening (either I'm doing something wrong or I'm misreading the documentation).

I have a cluster where I have 2 physical machines. On each physical machine I launch multiple data nodes. I'm configuring indexes to have 5 primary shards and 1 replica.

I've configured awareness to be on the physical host name, set the physical host name on each data node, and listed all the physical host names in the forced awareness arg.

When I create an index, I can see the primary shard and replica shard always assigned to nodes that reside on different physical machines.

However, if I change the replica count to 2, the cluster assigns both replicas. My understanding from the docs is that the 2nd replica shouldn't be assigned because there there are only 2 unique forced attributes, so one replica shard should go unassigned.

Snippet of cluster config (names replaced):

"settings": {
  "cluster": {
    "name": "myCluster",
    "routing": {
    "allocation": {
      "awareness": {
        "attributes": "host",
        "force": {
          "host": {
            "values": "physicalHost1,physicalHost2"
          }
        }
      }
    }
  }
},
"node": {
  "name": "physicalHost1.data7",
  "attr": {
    "host": "physicalHost1"
  },
  "data": "true",
  "ingest": "true",
  "master": "false"
}

Similarly, if I change replica count to 3, it assigns all 3, but it does ensure that between the 4 copies (1 primary and 3 replicas), 2 copies end up on 1 physical host, and 2 on the other.

Does forced awareness only ensure that 1 replica ends up on a node that has different properties? (and any additional replicas get distributed ignoring the properties?)

[ywelsch (Yannick Welsch)]

Awareness and forced awareness is a bit more general: It considers the number of distinct attribute values (2 in your case) and the number of expected shard copies (1 + number_of_replicas), and then ensures that each attribute gets its fair share of shard copies by dividing the number of shard copies by the number of distinct attribute values, which yields the maximum number of shard copies that it allows per attribute. For example, 4 shard copies for 2 distinct attributes means that the system will allow allocating 2 shard copies per attribute. Note that if the remainder of the division is > 0, it allows one extra shard copy to be allocated to the node. For example, 5 shard copies for 2 distinct attributes means that the system will allow 3 shard copies to be allocated per attribute.

[priteshkp (Pritesh Patel)]

Thanks @ywelsch, your description is very clear. That would be very useful to document on this page if you can:
https://www.elastic.co/guide/en/elasticsearch/reference/current/allocation-awareness.html