Forced-awareness in Elastic 8.0.1

Elastic Stack Elasticsearch
1

Hi
I have a question: how I can ensure shard replication for to scatter shards to other hosts?,
My Elastic cluster consist from 3 host (under 51 data nodes)
As You can see
Elasticsearch has made replicat on the same host, it doesn't make sens

logstash-data-log-2022.04.06 7 p STARTED 14122382 7.7gb 10.0.9.174 es_data_ssd_3_2
//last number in my case<- indicates host
logstash-data-log-2022.04.06 7 r STARTED 14118343 7.3gb 10.0.9.164 es_data_ssd_4_2
//so I have a primary and replicat on the same host, which is not intended

logstash-data-log-2022.04.06 7 p STARTED 14122382 7.7gb 10.0.9.174 es_data_ssd_3_2
logstash-data-log-2022.04.06 7 r STARTED 14118343 7.3gb 10.0.9.164 es_data_ssd_4_2
logstash-data-log-2022.04.06 5 p STARTED 14119279 7.3gb 10.0.9.203 es_data_ssd_2_3
logstash-data-log-2022.04.06 5 r STARTED 14119279 7.6gb 10.0.9.181 es_data_ssd_3_3
logstash-data-log-2022.04.06 4 p STARTED 14109308 7.6gb 10.0.9.135 es_data_ssd_5_1
logstash-data-log-2022.04.06 4 r STARTED 14113259 10.3gb 10.0.9.199 es_data_ssd_5_3
logstash-data-log-2022.04.06 6 p STARTED 14113733 7.7gb 10.0.9.207 es_data_ssd_1_3
logstash-data-log-2022.04.06 6 r STARTED 14116927 7.7gb 10.0.9.125 es_data_ssd_4_1
logstash-data-log-2022.04.06 2 p STARTED 14118753 7.7gb 10.0.9.146 es_data_ssd_5_2
logstash-data-log-2022.04.06 2 r STARTED 14116560 7.7gb 10.0.9.131 es_data_ssd_3_1
logstash-data-log-2022.04.06 1 r STARTED 14117532 7.3gb 10.0.9.119 es_data_ssd_2_1
logstash-data-log-2022.04.06 1 p STARTED 14118898 7.4gb 10.0.9.170 es_data_ssd_2_2
logstash-data-log-2022.04.06 3 r STARTED 14111757 7.3gb 10.0.9.195 es_data_ssd_4_3
logstash-data-log-2022.04.06 3 p STARTED 14112069 7.6gb 10.0.9.137 es_data_ssd_1_1
logstash-data-log-2022.04.06 0 p STARTED 14115341 7.3gb 10.0.9.174 es_data_ssd_3_2
logstash-data-log-2022.04.06 0 r STARTED 14112860 7.6gb 10.0.9.158 es_data_ssd_1_2

should I follow by cluster.routing.allocation.same_shard.host

2

One thing it's worth to add that:

below nodes were placed on the same one host You can see the different host IP because all these nodes were managed by docker swarm on 3 site (bare metal hosts)

so in this case how it should be manged by "cluster.routing.allocation.same_shard.host"


      "name" : "es_data_ssd_5_1",
      "transport_address" : "10.0.9.135:9300",
      "host" : "10.0.9.135",
      "ip" : "10.0.9.135:9300",
      "roles" : [
        "data_content",
        "data_hot"
      ],


    "name" : "es_data_ssd_4_1",
      "transport_address" : "10.0.9.125:9300",
      "host" : "10.0.9.125",
      "ip" : "10.0.9.125:9300",
      "roles" : [
        "data_content",
        "data_hot"
      ],

      "name" : "es_data_ssd_3_1",
      "transport_address" : "10.0.9.131:9300",
      "host" : "10.0.9.131",
      "ip" : "10.0.9.131:9300",
      "roles" : [
        "data_content",
        "data_hot"
      ],

    "name" : "es_data_ssd_2_1",
      "transport_address" : "10.0.9.119:9300",
      "host" : "10.0.9.119",
      "ip" : "10.0.9.119:9300",
      "roles" : [
        "data_content",
        "data_hot"
      ],

      "name" : "es_data_ssd_1_1",
      "transport_address" : "10.0.9.137:9300",
      "host" : "10.0.9.137",
      "ip" : "10.0.9.137:9300",
      "roles" : [
        "data_content",
        "data_hot"
      ],
3

Elasticsearch has no information about your physical hosts, only the nodes running on them.

If the nodes have different ip address, they will be seen as different hosts, so there is nothing that forbids elasticsearch to allocate a primary shard and a replica in nodes that are in the same physical host but have different ip addresses.

The cluster.routing.allocation.same_shard.host also will take the same approach, it will take the ip address of the host for the allocation.

From the documentation:

If true , forbids multiple copies of a shard from being allocated to distinct nodes on the same host, i.e. which have the same network address. Defaults to false , meaning that copies of a shard may sometimes be allocated to nodes on the same host. This setting is only relevant if you run multiple nodes on each host.

What you need to do is use the shard allocation awereness, which is based on a custom attribute to tell elasticsearch to take the physical host in consideration when allocating shards.

You will need to create some attribute to differentiate the physical hosts you have, like the example in the documentation you could use rack_id as the attribute and use rack_one, rack_two and rack_three as the values.

I do not use docker, so I have no idea how you will automate it, but this is the way to go.

1 Like
4

Starting with ECK 2.0 the operator can make Kubernetes Node labels available as Pod annotations. It can be used to make information, such as logical failure domains, available in a running Pod. Combined with Elasticsearch shard allocation awareness and Kubernetes topology spread constraints, you can create an availability zone-aware Elasticsearch cluster.

1 Like
5

Thanks for point out, but for more clarify, in that case should I add to all data node
node.attr.rack_id:
and to particular master node cluster.routing.allocation.awareness.attributes: rack_id
???

6

well so can I also use such config over zone??? Can I assume that zons are my physical hosts?

master node 1
cluster.routing.allocation.awareness.attributes: zone
cluster.routing.allocation.awareness.force.zone.values: zone1,zone2,zone3
node.attr.zone: zone1


master node 2
cluster.routing.allocation.awareness.attributes: zone
cluster.routing.allocation.awareness.force.zone.values: zone1,zone2,zone3
node.attr.zone: zone2


master node 3
cluster.routing.allocation.awareness.attributes: zone
cluster.routing.allocation.awareness.force.zone.values: zone1,zone2,zone3
node.attr.zone: zone3


zone1
data node 1_1
node.attr.zone: zone1

data node 1_2
node.attr.zone: zone1

data node 1_3
node.attr.zone: zone1


zone2
data node 2_1
node.attr.zone: zone2

data node 2_2
node.attr.zone: zone2

data node 2_3
node.attr.zone: zone2


zone3
data node 3_1
node.attr.zone: zone3

data node 3_2
node.attr.zone: zone3

data node 3_3
node.attr.zone: zone3
7

I would also like to bring back the question asked when this case ES awareness zones/racks

And I am not sure what is the difference between this:
e.g
node.rack_id: rack_1
cluster.routing.allocation.awareness.attributes: rack_id

or
node.zone: zone1
cluster.routing.allocation.awareness.attributes: zone

Do they serve the same purpose?

  1. cluster.routing.allocation.awareness.attributes: rack_id,zone
    How does this work?
  2. for forced awareness:
    cluster.routing.allocation.awareness.force.zone.values: zone1,zone2

I can see how this works, but is this applicable to only "zones"?
or you can do:
cluster.routing.allocation.awareness.force.rack_id.values: rack_1,rack_2?

8

I've tried below settings but something goes wrong container was restarted in loop

version: "3.7"
  
services:
  es_master_1_3:
    image: priv_repo/elk-docker/elasticsearch/elasticsearch:8.1.0
    container_name: es_master_1_3
    environment:
      - node.name=es_master_1_3
      - xpack.security.enabled=true
      - xpack.ml.enabled=false
      - xpack.security.http.ssl.enabled=false
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=basic
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.certificate=certs/es_master_1_3/es_master_1_3.crt
      - xpack.security.transport.ssl.key=certs/es_master_1_3/es_master_1_3.key
      - ELASTIC_USERNAME=elastic
      - ELASTIC_PASSWORD= 
      - cluster.name=elk_cluster
      - discovery.seed_hosts=es_master_1_1,es_master_2_1,es_master_1_2,es_master_2_2,es_master_2_3,es_data_ssd_3_1_ingest,es_data_ssd_1_1,es_data_ssd_2_1,es_data_ssd_3_1,es_data_ssd_4_1,es_data_ssd_5_1,es_data_hdd_1_1,es_data_hdd_2_1,es_data_hdd_3_1,es_data_hdd_4_1,es_data_hdd_5_1,es_data_hdd_6_1,es_data_hdd_7_1,es_data_hdd_8_1,es_data_hdd_9_1,es_data_ssd_3_2_ingest,es_data_ssd_1_2,es_data_ssd_2_2,es_data_ssd_3_2,es_data_ssd_4_2,es_data_ssd_5_2,es_data_hdd_1_2,es_data_hdd_2_2,es_data_hdd_3_2,es_data_hdd_4_2,es_data_hdd_5_2,es_data_hdd_6_2,es_data_hdd_7_2,es_data_hdd_8_2,es_data_hdd_9_2,es_data_ssd_3_3_ingest,es_data_ssd_1_3,es_data_ssd_2_3,es_data_ssd_3_3,es_data_ssd_4_3,es_data_ssd_5_3,es_data_hdd_1_3,es_data_hdd_2_3,es_data_hdd_3_3,es_data_hdd_4_3,es_data_hdd_5_3,es_data_hdd_6_3,es_data_hdd_7_3,es_data_hdd_8_3,es_data_hdd_9_3
      - cluster.initial_master_nodes=es_master_1_1,es_master_2_1,es_master_1_2,es_master_2_2,es_master_1_3,es_master_2_3
      - bootstrap.memory_lock=true
      - ingest.geoip.downloader.enabled=false
      - node.roles=master
      - cluster.routing.allocation.awareness.attributes= zone
      - cluster.routing.allocation.awareness.force.zone.values= zone1,zone2,zone3
      - node.attr.zone= zone3
9

next question is: where does container store these environments, I saw that this syntax is a little bit different what we used in Elasticsearch.yml

10

internal bump

11

There is no difference, you choose the name of the attribute you want, it doe not need to be rack_id or zone, it can be something else.

You can have multiple customs attributes and choose one or more to use forced awereness.

I'm sorry, I do not run elasticsearch on containers, so I can not help here.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.