# Forced-awareness in Elastic 8.0.1

**URL:** <https://discuss.elastic.co/t/forced-awareness-in-elastic-8-0-1/301738>\
**Category:** Elasticsearch\
**Created:** [April 6, 2022, 10:25am UTC](https://discuss.elastic.co/t/forced-awareness-in-elastic-8-0-1/301738 "2022-04-06T10:25:07Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\
**Post date:** [April 6, 2022, 10:25am UTC](https://discuss.elastic.co/t/forced-awareness-in-elastic-8-0-1/301738/1 "2022-04-06T10:25:07Z")

</div>

Hi  
I have a question: how I can ensure shard replication for to scatter shards to other hosts?,  
My Elastic cluster consist from 3 host (under 51 data nodes)  
As You can see  
Elasticsearch has made replicat on the same host, it doesn't make sens

`logstash-data-log-2022.04.06 7 p STARTED 14122382 7.7gb 10.0.9.174 es_data_ssd_3_2`  
//last number in my case\<- indicates host  
`logstash-data-log-2022.04.06 7 r STARTED 14118343 7.3gb 10.0.9.164 es_data_ssd_4_2`  
//so I have a primary and replicat on the same host, which is not intended

```auto
logstash-data-log-2022.04.06 7 p STARTED 14122382 7.7gb 10.0.9.174 es_data_ssd_3_2
logstash-data-log-2022.04.06 7 r STARTED 14118343 7.3gb 10.0.9.164 es_data_ssd_4_2
logstash-data-log-2022.04.06 5 p STARTED 14119279 7.3gb 10.0.9.203 es_data_ssd_2_3
logstash-data-log-2022.04.06 5 r STARTED 14119279 7.6gb 10.0.9.181 es_data_ssd_3_3
logstash-data-log-2022.04.06 4 p STARTED 14109308 7.6gb 10.0.9.135 es_data_ssd_5_1
logstash-data-log-2022.04.06 4 r STARTED 14113259 10.3gb 10.0.9.199 es_data_ssd_5_3
logstash-data-log-2022.04.06 6 p STARTED 14113733 7.7gb 10.0.9.207 es_data_ssd_1_3
logstash-data-log-2022.04.06 6 r STARTED 14116927 7.7gb 10.0.9.125 es_data_ssd_4_1
logstash-data-log-2022.04.06 2 p STARTED 14118753 7.7gb 10.0.9.146 es_data_ssd_5_2
logstash-data-log-2022.04.06 2 r STARTED 14116560 7.7gb 10.0.9.131 es_data_ssd_3_1
logstash-data-log-2022.04.06 1 r STARTED 14117532 7.3gb 10.0.9.119 es_data_ssd_2_1
logstash-data-log-2022.04.06 1 p STARTED 14118898 7.4gb 10.0.9.170 es_data_ssd_2_2
logstash-data-log-2022.04.06 3 r STARTED 14111757 7.3gb 10.0.9.195 es_data_ssd_4_3
logstash-data-log-2022.04.06 3 p STARTED 14112069 7.6gb 10.0.9.137 es_data_ssd_1_1
logstash-data-log-2022.04.06 0 p STARTED 14115341 7.3gb 10.0.9.174 es_data_ssd_3_2
logstash-data-log-2022.04.06 0 r STARTED 14112860 7.6gb 10.0.9.158 es_data_ssd_1_2

```

should I follow by `cluster.routing.allocation.same_shard.host`

---

<div class="post-metadata">

**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\
**Post date:** [April 6, 2022, 12:50pm UTC](https://discuss.elastic.co/t/forced-awareness-in-elastic-8-0-1/301738/2 "2022-04-06T12:50:48Z")

</div>

One thing it's worth to add that:

below nodes were placed on the same one host You can see the different host IP because all these nodes were managed by docker swarm on 3 site (bare metal hosts)

so in this case how it should be manged by "cluster.routing.allocation.same\_shard.host"

```auto

      "name" : "es_data_ssd_5_1",
      "transport_address" : "10.0.9.135:9300",
      "host" : "10.0.9.135",
      "ip" : "10.0.9.135:9300",
      "roles" : [
        "data_content",
        "data_hot"
      ],

```

```auto

    "name" : "es_data_ssd_4_1",
      "transport_address" : "10.0.9.125:9300",
      "host" : "10.0.9.125",
      "ip" : "10.0.9.125:9300",
      "roles" : [
        "data_content",
        "data_hot"
      ],

```

```auto
      "name" : "es_data_ssd_3_1",
      "transport_address" : "10.0.9.131:9300",
      "host" : "10.0.9.131",
      "ip" : "10.0.9.131:9300",
      "roles" : [
        "data_content",
        "data_hot"
      ],

```

```auto
    "name" : "es_data_ssd_2_1",
      "transport_address" : "10.0.9.119:9300",
      "host" : "10.0.9.119",
      "ip" : "10.0.9.119:9300",
      "roles" : [
        "data_content",
        "data_hot"
      ],

```

```auto
      "name" : "es_data_ssd_1_1",
      "transport_address" : "10.0.9.137:9300",
      "host" : "10.0.9.137",
      "ip" : "10.0.9.137:9300",
      "roles" : [
        "data_content",
        "data_hot"
      ],

```

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [April 6, 2022, 1:59pm UTC](https://discuss.elastic.co/t/forced-awareness-in-elastic-8-0-1/301738/3 "2022-04-06T13:59:15Z")

</div>

Elasticsearch has no information about your physical hosts, only the nodes running on them.

If the nodes have different ip address, they will be seen as different hosts, so there is nothing that forbids elasticsearch to allocate a primary shard and a replica in nodes that are in the same physical host but have different ip addresses.

The `cluster.routing.allocation.same_shard.host` also will take the same approach, it will take the ip address of the host for the allocation.

From the documentation:

> If `true` , forbids multiple copies of a shard from being allocated to distinct nodes on the same host, i.e. **which have the same network address**. Defaults to `false` , meaning that copies of a shard may sometimes be allocated to nodes on the same host. **This setting is only relevant if you run multiple nodes on each host.**

What you need to do is use the [shard allocation awereness](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cluster.html#shard-allocation-awareness), which is based on a custom attribute to tell elasticsearch to take the physical host in consideration when allocating shards.

You will need to create some attribute to differentiate the physical hosts you have, like the example in the documentation you could use `rack_id` as the attribute and use `rack_one`, `rack_two` and `rack_three` as the values.

I do not use docker, so I have no idea how you will automate it, but this is the way to go.

---

<div class="post-metadata">

**Author:** ![Thibault\_Richard](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thibault_richard/32/50513_2.png) [@Thibault\_Richard](https://discuss.elastic.co/u/Thibault_Richard)\
**Post date:** [April 6, 2022, 4:04pm UTC](https://discuss.elastic.co/t/forced-awareness-in-elastic-8-0-1/301738/4 "2022-04-06T16:04:54Z")

</div>

Starting with ECK 2.0 the operator can make Kubernetes Node labels available as Pod annotations. It can be used to make information, such as logical failure domains, available in a running Pod. Combined with [Elasticsearch shard allocation awareness](https://www.elastic.co/guide/en/elasticsearch/reference/current/allocation-awareness.html#allocation-awareness) and [Kubernetes topology spread constraints](https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/), you can create an availability zone-aware Elasticsearch cluster.

> **[Advanced Elasticsearch node scheduling | Elastic Cloud on Kubernetes \[2.1\] |...](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-advanced-node-scheduling.html#k8s-availability-zone-awareness)**

---

<div class="post-metadata">

**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\
**Post date:** [April 6, 2022, 4:53pm UTC](https://discuss.elastic.co/t/forced-awareness-in-elastic-8-0-1/301738/5 "2022-04-06T16:53:41Z")

</div>

> [@leandrojmp](#):
>
> you could use `rack_id` as the attribute and use `rack_one` , `rack_two` and `rack_three` as the values.

Thanks for point out, but for more clarify, in that case should I add to all data node  
`node.attr.rack_id:`  
and to particular master node `cluster.routing.allocation.awareness.attributes: rack_id`  
???

---

<div class="post-metadata">

**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\
**Post date:** [April 6, 2022, 7:27pm UTC](https://discuss.elastic.co/t/forced-awareness-in-elastic-8-0-1/301738/6 "2022-04-06T19:27:11Z")

</div>

well so can I also use such config over zone??? Can I assume that zons are my physical hosts?

```auto
master node 1
cluster.routing.allocation.awareness.attributes: zone
cluster.routing.allocation.awareness.force.zone.values: zone1,zone2,zone3
node.attr.zone: zone1

master node 2
cluster.routing.allocation.awareness.attributes: zone
cluster.routing.allocation.awareness.force.zone.values: zone1,zone2,zone3
node.attr.zone: zone2

master node 3
cluster.routing.allocation.awareness.attributes: zone
cluster.routing.allocation.awareness.force.zone.values: zone1,zone2,zone3
node.attr.zone: zone3

zone1
data node 1_1
node.attr.zone: zone1

data node 1_2
node.attr.zone: zone1

data node 1_3
node.attr.zone: zone1

zone2
data node 2_1
node.attr.zone: zone2

data node 2_2
node.attr.zone: zone2

data node 2_3
node.attr.zone: zone2

zone3
data node 3_1
node.attr.zone: zone3

data node 3_2
node.attr.zone: zone3

data node 3_3
node.attr.zone: zone3

```

---

<div class="post-metadata">

**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\
**Post date:** [April 6, 2022, 7:43pm UTC](https://discuss.elastic.co/t/forced-awareness-in-elastic-8-0-1/301738/7 "2022-04-06T19:43:42Z")

</div>

I would also like to bring back the question asked when this case [ES awareness zones/racks](https://discuss.elastic.co/t/es-awareness-zones-racks/116578)

And I am not sure what is the difference between this:  
e.g  
node.rack\_id: rack\_1  
cluster.routing.allocation.awareness.attributes: rack\_id

or  
node.zone: zone1  
cluster.routing.allocation.awareness.attributes: zone

Do they serve the same purpose?

1. cluster.routing.allocation.awareness.attributes: rack\_id,zone  
How does this work?
2. for forced awareness:  
cluster.routing.allocation.awareness.force.zone.values: zone1,zone2

I can see how this works, but is this applicable to only "zones"?  
or you can do:  
cluster.routing.allocation.awareness.force.rack\_id.values: rack\_1,rack\_2?

---

<div class="post-metadata">

**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\
**Post date:** [April 6, 2022, 9:27pm UTC](https://discuss.elastic.co/t/forced-awareness-in-elastic-8-0-1/301738/8 "2022-04-06T21:27:04Z")

</div>

I've tried below settings but something goes wrong container was restarted in loop

```auto
version: "3.7"
  
services:
  es_master_1_3:
    image: priv_repo/elk-docker/elasticsearch/elasticsearch:8.1.0
    container_name: es_master_1_3
    environment:
      - node.name=es_master_1_3
      - xpack.security.enabled=true
      - xpack.ml.enabled=false
      - xpack.security.http.ssl.enabled=false
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=basic
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.certificate=certs/es_master_1_3/es_master_1_3.crt
      - xpack.security.transport.ssl.key=certs/es_master_1_3/es_master_1_3.key
      - ELASTIC_USERNAME=elastic
      - ELASTIC_PASSWORD= 
      - cluster.name=elk_cluster
      - discovery.seed_hosts=es_master_1_1,es_master_2_1,es_master_1_2,es_master_2_2,es_master_2_3,es_data_ssd_3_1_ingest,es_data_ssd_1_1,es_data_ssd_2_1,es_data_ssd_3_1,es_data_ssd_4_1,es_data_ssd_5_1,es_data_hdd_1_1,es_data_hdd_2_1,es_data_hdd_3_1,es_data_hdd_4_1,es_data_hdd_5_1,es_data_hdd_6_1,es_data_hdd_7_1,es_data_hdd_8_1,es_data_hdd_9_1,es_data_ssd_3_2_ingest,es_data_ssd_1_2,es_data_ssd_2_2,es_data_ssd_3_2,es_data_ssd_4_2,es_data_ssd_5_2,es_data_hdd_1_2,es_data_hdd_2_2,es_data_hdd_3_2,es_data_hdd_4_2,es_data_hdd_5_2,es_data_hdd_6_2,es_data_hdd_7_2,es_data_hdd_8_2,es_data_hdd_9_2,es_data_ssd_3_3_ingest,es_data_ssd_1_3,es_data_ssd_2_3,es_data_ssd_3_3,es_data_ssd_4_3,es_data_ssd_5_3,es_data_hdd_1_3,es_data_hdd_2_3,es_data_hdd_3_3,es_data_hdd_4_3,es_data_hdd_5_3,es_data_hdd_6_3,es_data_hdd_7_3,es_data_hdd_8_3,es_data_hdd_9_3
      - cluster.initial_master_nodes=es_master_1_1,es_master_2_1,es_master_1_2,es_master_2_2,es_master_1_3,es_master_2_3
      - bootstrap.memory_lock=true
      - ingest.geoip.downloader.enabled=false
      - node.roles=master
      - cluster.routing.allocation.awareness.attributes= zone
      - cluster.routing.allocation.awareness.force.zone.values= zone1,zone2,zone3
      - node.attr.zone= zone3

```

---

<div class="post-metadata">

**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\
**Post date:** [April 7, 2022, 6:08am UTC](https://discuss.elastic.co/t/forced-awareness-in-elastic-8-0-1/301738/9 "2022-04-07T06:08:14Z")

</div>

next question is: where does container store these environments, I saw that this syntax is a little bit different what we used in Elasticsearch.yml

---

<div class="post-metadata">

**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\
**Post date:** [April 13, 2022, 6:16pm UTC](https://discuss.elastic.co/t/forced-awareness-in-elastic-8-0-1/301738/10 "2022-04-13T18:16:51Z")

</div>

internal bump

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [April 14, 2022, 2:44am UTC](https://discuss.elastic.co/t/forced-awareness-in-elastic-8-0-1/301738/11 "2022-04-14T02:44:57Z")

</div>

> [@INS](#):
>
> And I am not sure what is the difference between this:  
> e.g  
> node.rack\_id: rack\_1  
> cluster.routing.allocation.awareness.attributes: rack\_id
> 
> or  
> node.zone: zone1  
> cluster.routing.allocation.awareness.attributes: zone

There is no difference, you choose the name of the attribute you want, it doe not need to be `rack_id` or `zone`, it can be something else.

You can have multiple customs attributes and choose one or more to use forced awereness.

> [@INS](#):
>
> next question is: where does container store these environments, I saw that this syntax is a little bit different what we used in Elasticsearch.yml

I'm sorry, I do not run elasticsearch on containers, so I can not help here.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 12, 2022, 2:45am UTC](https://discuss.elastic.co/t/forced-awareness-in-elastic-8-0-1/301738/12 "2022-05-12T02:45:30Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.