# Same shards on different physicals servers

**URL:** <https://discuss.elastic.co/t/same-shards-on-different-physicals-servers/346768>\
**Category:** Elasticsearch\
**Created:** [November 9, 2023, 9:09am UTC](https://discuss.elastic.co/t/same-shards-on-different-physicals-servers/346768 "2023-11-09T09:09:46Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![daniela09](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daniela09/32/120636_2.png) [@daniela09](https://discuss.elastic.co/u/daniela09)\
**Post date:** [November 9, 2023, 9:09am UTC](https://discuss.elastic.co/t/same-shards-on-different-physicals-servers/346768/1 "2023-11-09T09:09:46Z")

</div>

Hi I have deployed EFK stack on Kubernetes cluster, I have 3 nodes that have both roles data and master, the 3 Elasticsearch nodes are on 3 different Kubernetes nodes, but the Kubernetes nodes are on 2 different physical servers, so that means that 2 of the 3 Elasticsearch nodes will always be on the same physical server, so I want to make sure that replica shards are on node that is on different physical server then the primary shard. Is there a way I can do this?

I saw there is way to set **cluster.routing.allocation.same\_shard.host: true,** but from the response from GET \_nodes/stats, I see that the 3 Elasticsearch nodes have different IP address for host, that is because the 3 Kubernetes nodes that the Elasticsearch nodes are on have different IP addresses. So I don't think this will work because the value for host will always be different.

---

<div class="post-metadata">

**Author:** ![bunste](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bunste/32/95865_2.png) [@bunste](https://discuss.elastic.co/u/bunste)\
**Post date:** [November 9, 2023, 9:26am UTC](https://discuss.elastic.co/t/same-shards-on-different-physicals-servers/346768/2 "2023-11-09T09:26:04Z")

</div>

This should work if you add the following settings to `elasticsearch.yml` on all nodes:

```auto
node.attr.machine: {{ physical_host }}
cluster.routing.allocation.awareness.attributes: machine

```

For **{{ physical\_host }}** you put in some identifier for your two servers (like _k8s-host-01_, _k8s-host-02_)

This way elasticsearch won't put replicas on the same physical host as their primary shards.

Check out the docs on [Shard Allocation Awareness](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cluster.html#shard-allocation-awareness).

---

<div class="post-metadata">

**Author:** ![daniela09](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daniela09/32/120636_2.png) [@daniela09](https://discuss.elastic.co/u/daniela09)\
**Post date:** [November 9, 2023, 10:44am UTC](https://discuss.elastic.co/t/same-shards-on-different-physicals-servers/346768/3 "2023-11-09T10:44:01Z")

</div>

If I add labels to my Kubernetes nodes, on 2 of them I add label server=host1 and on 1 of them I add label server=host2 and I add this in the elasticsearch.yaml:

```auto
node.attr.server: host1, host2
cluster.routing.allocation.awareness.attributes: server 

```

Will this work, would Elastic nodes be able to see the labels on the Kubernetes nodes they are on?  
Or is there any way Elastic nodes can see labels on Kubernetes nodes they are on?

---

<div class="post-metadata">

**Author:** ![bunste](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bunste/32/95865_2.png) [@bunste](https://discuss.elastic.co/u/bunste)\
**Post date:** [November 9, 2023, 11:01am UTC](https://discuss.elastic.co/t/same-shards-on-different-physicals-servers/346768/4 "2023-11-09T11:01:22Z")

</div>

That's not quite what I had in mind. You only need to edit the Elasticsearch config of the three instances - nothing else.

Assuming you have 2 Kubernetes nodes on `host1` and 1 Kubernetes node on `host2`. Then configure the two Elasticsearch instances on `host1` as follows:

```auto
node.attr.machine: host1
cluster.routing.allocation.awareness.attributes: machine

```

And configure the third Elasticsearch instance on `host2` as follows:

```auto
node.attr.machine: host2
cluster.routing.allocation.awareness.attributes: machine

```

That should be all there is to it.

---

<div class="post-metadata">

**Author:** ![daniela09](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daniela09/32/120636_2.png) [@daniela09](https://discuss.elastic.co/u/daniela09)\
**Post date:** [November 9, 2023, 12:10pm UTC](https://discuss.elastic.co/t/same-shards-on-different-physicals-servers/346768/5 "2023-11-09T12:10:14Z")

</div>

host1 and host2 can be any names, when I will have

```auto
node.attr.machine: host1

```

on one Elasticsearch instance and

```auto
node.attr.machine: host2

```

on the other two Elasticsearch instances.

Elasticsearch will know which instances are on the same physical server and it will distribute the primary shard and its replica shard to be on Elasticsearch instances with different "node.attr.machine:" value?

---

<div class="post-metadata">

**Author:** ![bunste](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bunste/32/95865_2.png) [@bunste](https://discuss.elastic.co/u/bunste)\
**Post date:** [November 9, 2023, 1:52pm UTC](https://discuss.elastic.co/t/same-shards-on-different-physicals-servers/346768/6 "2023-11-09T13:52:49Z")

</div>

Yes, exactly. You can read more about how this works in the docs like I provided.

---

<div class="post-metadata">

**Author:** ![daniela09](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daniela09/32/120636_2.png) [@daniela09](https://discuss.elastic.co/u/daniela09)\
**Post date:** [November 9, 2023, 2:27pm UTC](https://discuss.elastic.co/t/same-shards-on-different-physicals-servers/346768/7 "2023-11-09T14:27:22Z")

</div>

One more question, when I do a rollout restart on Elasticsearch statefulset, or 2 of the 3 nodes crush they can be scheduled on different Kubernetes node then they are now, in that case there is a chance that the "node.attr.machine:" value won't be correct, the value may be same for 2 Elasticsearch nodes that are on different physical servers and in that case the primary and replica shard would be on Elasticsearch nodes that are on same physical server.

Is there a way to configure the "node.attr.machine:" value dynamically depending on what Kubernetes node they are on?

---

<div class="post-metadata">

**Author:** ![bunste](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bunste/32/95865_2.png) [@bunste](https://discuss.elastic.co/u/bunste)\
**Post date:** [November 9, 2023, 4:01pm UTC](https://discuss.elastic.co/t/same-shards-on-different-physicals-servers/346768/8 "2023-11-09T16:01:04Z")

</div>

If you can ensure that an environment variable exists on the Kubernetes nodes that identifies the physical server (e.g. HOST), you can use this in your Elasticsearch config:

```
node.attr.machine: ${HOST}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 7, 2023, 4:01pm UTC](https://discuss.elastic.co/t/same-shards-on-different-physicals-servers/346768/9 "2023-12-07T16:01:17Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.