Question shard allocation awareness?

Elastic Stack Elasticsearch
1

Enabled shard allocation awareness/forced-awareness in the cluster, right now enabled for two zones

"cluster.routing.allocation.awareness.attributes" : "rack_id" "cluster.routing.allocation.awareness.force.rack_id.values" : "us-east-1a,us-east-1b"
Can I have more values for forced-awareness than enabled racks in my cluster,

for example cluster with two availability zone(based on replica config), us-east-1a,us-east-1b

but in future, we may split the cluster into three availability zone,

can I add all the possible values of attribute rack_id to forced-awareness config, even though clusters are running in two racks/ availability zone

cluster.routing.allocation.awareness.force.rack_id.values" : "us-east-1a,us-east-1b,us-east-1c,us-east-1d" something like this.

I enabled it in ES test cluster, I'm not seeing any issues, but wanted to know if there are any issues because of this setup?.

2

It's not a great idea, but I don't think you'll see any difference if all your indices have number_of_replicas: 1. If you have any indices with more replicas then they will not be fully allocated and your cluster health will go yellow.

I would recommend waiting until you have split the cluster across more zones. It is a simple change to adjust the awareness configuration via the API once you have more zones:

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.awareness.force.rack_id.values": "us-east-1a,us-east-1b,us-east-1c,us-east-1d"
  }
}

Therefore I don't see the benefit in setting this value any earlier.

3

so I use a template for installing the elastic search, if I make it static I don't have to change it every time I deploy a new cluster and also can be deployed in any availability zones.

One more question, after the shard allocation awareness in a cluster of rack 3,

let's say I have one primary and 2 replicas, will the primary will be allocated in a single rack or it will be allocated in multiple racks

4

I'm not sure I'm understanding the question. The primary is just another shard, each shard is allocated to one node and each node is in one rack so yes the primary will be in one rack.

5

Let me explain it in little bit detail,

suppose I created an index with 5 shards and 2 replica in a shard allocation awareness enabled cluster [3 racks/3 nodes]

so will it allocate primary shard in first node and replica1 in second node and replica2 in third node or it will be a mix of primary/replica in all nodes

image

Is this an expected behaviour in shard allocation awarness enabled cluster?.

thanks in advance

6

I see, you said "one primary" but now you are asking about the case with five primaries. Yes, they might well be spread out across the zones.

7

One more question , I have enabled cluster.routing.allocation.awareness.attributes for three racks(or three aws availability zone), and created an indices with default configuration(5 shards primary and 1 replica)

How will be allocation awareness works for this case will primary and replica will be in two racks or will it split into three racks ?.

8

Allocation awareness applies independently to each primary/replica pair. So primary and replica will be in two racks.

9

This is what, I'm trying to explain

image
image.png721×363 18.6 KB
,

so when I try to create an index with number of replicas 1 in a cluster where allocation awareness is enabled for three racks/availabilty zones, the shard allocation is not based on racks, it is trying to allocate in all the avaiilable racks.

but if the number of primary + replica equal to available racks, it works fine

Is this an expected behaviour?

10

I might be misinterpreting that graphic but it looks reasonable to me. Can you share the output of GET _cat/shards for the same situation? What exactly do you expect to be different with the allocation of rack1?

11

Output of GET _cat/shards

.kibana                         0 p STARTED    2   8.6kb 172.16.254.69  es-test-rack-master-b
rack1                           2 r STARTED    0    261b 172.16.254.69  es-test-rack-master-b
rack1                           2 p STARTED    0    261b 172.16.252.222 es-test-rack-master-a
rack1                           1 r STARTED    0    261b 172.16.251.47  es-test-rack-master-c
rack1                           1 p STARTED    0    261b 172.16.254.69  es-test-rack-master-b
rack1                           3 p STARTED    0    261b 172.16.251.47  es-test-rack-master-c
rack1                           3 r STARTED    0    261b 172.16.252.222 es-test-rack-master-a
rack1                           4 r STARTED    0    261b 172.16.251.47  es-test-rack-master-c
rack1                           4 p STARTED    0    261b 172.16.254.69  es-test-rack-master-b
rack1                           0 p STARTED    0    261b 172.16.251.47  es-test-rack-master-c
rack1                           0 r STARTED    0    261b 172.16.252.222 es-test-rack-master-a
.monitoring-es-6-2019.10.17     0 r STARTED 6783   3.6mb 172.16.251.47  es-test-rack-master-c
.monitoring-es-6-2019.10.17     0 p STARTED 6783   3.4mb 172.16.252.222 es-test-rack-master-a
rack2                           2 p STARTED    0    261b 172.16.251.47  es-test-rack-master-c
rack2                           2 r STARTED    0    261b 172.16.254.69  es-test-rack-master-b
rack2                           2 r STARTED    0    261b 172.16.252.222 es-test-rack-master-a
rack2                           1 r STARTED    0    261b 172.16.251.47  es-test-rack-master-c
rack2                           1 r STARTED    0    261b 172.16.254.69  es-test-rack-master-b
rack2                           1 p STARTED    0    261b 172.16.252.222 es-test-rack-master-a
rack2                           3 r STARTED    0    261b 172.16.251.47  es-test-rack-master-c
rack2                           3 p STARTED    0    261b 172.16.254.69  es-test-rack-master-b
rack2                           3 r STARTED    0    261b 172.16.252.222 es-test-rack-master-a
rack2                           4 r STARTED    0    261b 172.16.251.47  es-test-rack-master-c
rack2                           4 r STARTED    0    261b 172.16.254.69  es-test-rack-master-b
rack2                           4 p STARTED    0    261b 172.16.252.222 es-test-rack-master-a
rack2                           0 r STARTED    0    261b 172.16.251.47  es-test-rack-master-c
rack2                           0 p STARTED    0    261b 172.16.254.69  es-test-rack-master-b
rack2                           0 r STARTED    0    261b 172.16.252.222 es-test-rack-master-a
.monitoring-kibana-6-2019.10.17 0 p STARTED 6445 924.6kb 172.16.254.69  es-test-rack-master-b
.monitoring-kibana-6-2019.10.17 0 r STARTED 6445 910.7kb 172.16.252.222 es-test-rack-master-a

And for rack1 I thought it will be assigned in two racks since allocation awareness is enabled and number of replicas + primary equal to 2

12

I see. No, Elasticsearch will use all available nodes by default. If you want to restrict some indices to particular racks then you can use an allocation filter.

13

so the number of racks/availabilty zone should be equal to primary + replica?

for example if I created a cluster in three racks/availabilty zone with allocation awarenss support,

I should create indices with number of replicas 2, so that it will equally assigned on the racks

so is there any recomended method when using allocation awarenss support?
like any relation b/w number of replicas and racks enabled?

14

No, there's not really any recommendations in this area. Primary plus one replica spread (slightly unevenly) across three racks is a common setup.

15

To confirm, if I have primary + one replica it's better to run in two racks or if I have primary + 2 replica then use three racks so that shards allocate evenly?

16

As I said, there aren't really any recommendations in this area. It's common to have primary + one replica and three racks, which is neither of the alternatives you're asking about.

1 Like
17

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.