# Unable to setup Elasticsearch cluster mode

**URL:** <https://discuss.elastic.co/t/unable-to-setup-elasticsearch-cluster-mode/263987>\
**Category:** Elasticsearch\
**Created:** [February 11, 2021, 10:29am UTC](https://discuss.elastic.co/t/unable-to-setup-elasticsearch-cluster-mode/263987 "2021-02-11T10:29:13Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jose\_Angel\_Morena\_Si](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jose_angel_morena_si/32/76522_2.png) [@Jose\_Angel\_Morena\_Si](https://discuss.elastic.co/u/Jose_Angel_Morena_Si)\
**Post date:** [February 11, 2021, 10:29am UTC](https://discuss.elastic.co/t/unable-to-setup-elasticsearch-cluster-mode/263987/1 "2021-02-11T10:29:13Z")

</div>

Hello there,

I am trying to deploy Elasticsearch 7.10.2 in HA with three nodes, all of which play the roles: master, ingest and data.

In order to do so, I am using the following configuration in each of the nodes:  
NODE 1:

```auto
cluster.name: demo
node.name: node1
node.data: true
node.master: true
node.ingest: true
node.max_local_storage_nodes: 3
transport.tcp.port: 9300
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: false
network.host: 172.21.0.24
http.port: 9200
discovery.seed_hosts: ["172.21.0.24", "172.21.0.25","172.21.0.23"]
cluster.initial_master_nodes: ["172.21.0.24","172.21.0.25","172.21.0.23"]
discovery.zen.minimum_master_nodes: 2

```

NODE 2:

```auto
cluster.name: demo
node.name: node2
node.data: true
node.master: true
node.ingest: true
node.max_local_storage_nodes: 3
transport.tcp.port: 9300
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: false
network.host: 172.21.0.23
http.port: 9200
discovery.seed_hosts: ["172.21.0.24", "172.21.0.25","172.21.0.23"]
cluster.initial_master_nodes: ["172.21.0.24","172.21.0.25","172.21.0.23"]
discovery.zen.minimum_master_nodes: 2

```

NODE 3:

```auto
cluster.name: demo
node.name: node3
node.data: true
node.master: true
node.ingest: true
node.max_local_storage_nodes: 3
transport.tcp.port: 9300
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: false
network.host: 172.21.0.25
http.port: 9200
discovery.seed_hosts: ["172.21.0.24", "172.21.0.25","172.21.0.23"]
cluster.initial_master_nodes: ["172.21.0.24","172.21.0.25","172.21.0.23"]
discovery.zen.minimum_master_nodes: 2

```

When I curl one of the nodes to see what nodes belong to the cluster, there seems to be one node not being added:

```auto
[root@bastion elastic-playbooks]# ansible elastic -m shell -a "curl http://{{ inventory_hostname }}:9200/_cat/nodes? --silent"
172.21.0.23 | CHANGED | rc=0 >>
172.21.0.23 16 9 0 0.00 0.01 0.05 cdhilmrstw * node2
172.21.0.24 | CHANGED | rc=0 >>
172.21.0.25 41 9 0 0.01 0.09 0.08 cdhilmrstw - node3
172.21.0.24 39 9 0 0.01 0.06 0.06 cdhilmrstw * node1
172.21.0.25 | CHANGED | rc=0 >>
172.21.0.23 17 9 0 0.00 0.01 0.05 cdhilmrstw - node2
172.21.0.25 44 9 0 0.01 0.08 0.08 cdhilmrstw * node3

```

How can this be happening? Am I missing some important option?

Another doubt is that the `cluster_uuid` is different for each of the three nodes:

```auto
[root@bastion elastic-playbooks]# ansible elastic -m shell -a "curl http://{{ inventory_hostname }}:9200 --silent"

172.21.0.25 | CHANGED | rc=0 >>
{
  "name" : "node3",
  "cluster_name" : "demo",
  "cluster_uuid" : "BbwIsWoQTxGrhGoiSdPgLQ",
  "version" : {
    "number" : "7.10.2",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "747e1cc71def077253878a59143c1f785afa92b9",
    "build_date" : "2021-01-13T00:42:12.435326Z",
    "build_snapshot" : false,
    "lucene_version" : "8.7.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
172.21.0.23 | CHANGED | rc=0 >>
{
  "name" : "node2",
  "cluster_name" : "demo",
  "cluster_uuid" : "TJWYop8pR8mvlga4dVmf2g",
  "version" : {
    "number" : "7.10.2",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "747e1cc71def077253878a59143c1f785afa92b9",
    "build_date" : "2021-01-13T00:42:12.435326Z",
    "build_snapshot" : false,
    "lucene_version" : "8.7.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
172.21.0.24 | CHANGED | rc=0 >>
{
  "name" : "node1",
  "cluster_name" : "demo",
  "cluster_uuid" : "txrgSqiYRp-Oh-EtrZnWkw",
  "version" : {
    "number" : "7.10.2",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "747e1cc71def077253878a59143c1f785afa92b9",
    "build_date" : "2021-01-13T00:42:12.435326Z",
    "build_snapshot" : false,
    "lucene_version" : "8.7.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

```

Thanks in advance,

---

<div class="post-metadata">

**Author:** ![ylasri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ylasri/32/86120_2.png) [@ylasri](https://discuss.elastic.co/u/ylasri)\
**Post date:** [February 11, 2021, 11:43am UTC](https://discuss.elastic.co/t/unable-to-setup-elasticsearch-cluster-mode/263987/2 "2021-02-11T11:43:47Z")

</div>

Try this, The node names used in the `cluster.initial_master_nodes` list must exactly match the `node.name`

```auto
cluster.initial_master_nodes: ["node1", "node2", "node3"]

```

---

<div class="post-metadata">

**Author:** ![Jose\_Angel\_Morena\_Si](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jose_angel_morena_si/32/76522_2.png) [@Jose\_Angel\_Morena\_Si](https://discuss.elastic.co/u/Jose_Angel_Morena_Si)\
**Post date:** [February 11, 2021, 12:16pm UTC](https://discuss.elastic.co/t/unable-to-setup-elasticsearch-cluster-mode/263987/3 "2021-02-11T12:16:03Z")

</div>

Hello Yassine,

First of all thank you for your quick response! 🙂

I modified the configuration, now it looks like the following:

```auto
[root@bastion elastic-playbooks]# ansible elastic -m shell -a 'cat /etc/elasticsearch/elasticsearch.yml' 
172.21.0.25 | CHANGED | rc=0 >>
cluster.name: demo
node.name: es3
node.data: true
node.master: true
node.ingest: true
#node.max_local_storage_nodes: 3
transport.tcp.port: 9300
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: false
network.host: 172.21.0.25
http.port: 9200
discovery.seed_hosts: ["172.21.0.23", "172.21.0.24","172.21.0.25"]
cluster.initial_master_nodes: ["es1", "es2","es3"]
discovery.zen.minimum_master_nodes: 2
172.21.0.24 | CHANGED | rc=0 >>
cluster.name: demo
node.name: es2
node.data: true
node.master: true
node.ingest: true
#node.max_local_storage_nodes: 3
transport.tcp.port: 9300
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: false
network.host: 172.21.0.24
http.port: 9200
discovery.seed_hosts: ["172.21.0.23", "172.21.0.24","172.21.0.25"]
cluster.initial_master_nodes: ["es1", "es2","es3"]
discovery.zen.minimum_master_nodes: 2
172.21.0.23 | CHANGED | rc=0 >>
cluster.name: demo
node.name: es1
node.data: true
node.master: true
node.ingest: true
#node.max_local_storage_nodes: 3
transport.tcp.port: 9300
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: false
network.host: 172.21.0.23
http.port: 9200
discovery.seed_hosts: ["172.21.0.23", "172.21.0.24","172.21.0.25"]
cluster.initial_master_nodes: ["es1", "es2","es3"]
discovery.zen.minimum_master_nodes: 2

```

The problem is the following:

```auto
[root@bastion elastic-playbooks]# ansible elastic -m shell -a "curl http://{{ inventory_hostname }}:9200/_cat/nodes? --silent"
172.21.0.24 | CHANGED | rc=0 >>
172.21.0.23 7 9 0 0.06 0.03 0.05 cdhilmrstw - es1
172.21.0.24 59 9 0 0.00 0.03 0.05 cdhilmrstw * es2
172.21.0.25 8 9 0 0.00 0.01 0.05 cdhilmrstw - es3
172.21.0.25 | CHANGED | rc=0 >>
{"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}
172.21.0.23 | CHANGED | rc=0 >>
{"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}

```

How is it possible that the rest of the nodes: es2 and es3 cannot discover the masters? The port 9300 is accessible because there is no firewall, I already checked.

---

<div class="post-metadata">

**Author:** ![ylasri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ylasri/32/86120_2.png) [@ylasri](https://discuss.elastic.co/u/ylasri)\
**Post date:** [February 11, 2021, 12:55pm UTC](https://discuss.elastic.co/t/unable-to-setup-elasticsearch-cluster-mode/263987/4 "2021-02-11T12:55:19Z")

</div>

`transport.port` or `transport.tcp.port` ?

---

<div class="post-metadata">

**Author:** ![Jose\_Angel\_Morena\_Si](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jose_angel_morena_si/32/76522_2.png) [@Jose\_Angel\_Morena\_Si](https://discuss.elastic.co/u/Jose_Angel_Morena_Si)\
**Post date:** [February 11, 2021, 1:06pm UTC](https://discuss.elastic.co/t/unable-to-setup-elasticsearch-cluster-mode/263987/5 "2021-02-11T13:06:58Z")

</div>

Hello,

I have tried with the option `transport.port` and the result is the following:

```auto
[root@bastion elastic-playbooks]# ansible elastic -m shell -a "curl http://{{ inventory_hostname }}:9200/_cat/nodes? --silent"
172.21.0.24 | CHANGED | rc=0 >>
172.21.0.24 27 9 1 0.06 0.07 0.06 cdhilmrstw * es2
172.21.0.25 | CHANGED | rc=0 >>
172.21.0.25 26 9 1 0.14 0.09 0.07 cdhilmrstw * es3
172.21.0.23 | CHANGED | rc=0 >>
172.21.0.23 25 9 1 0.05 0.06 0.06 cdhilmrstw * es1

```

After trying setting both options the result was the same.

Do i have to reboot the Elasticsearch nodes one by one?

Thanks for your help,

---

<div class="post-metadata">

**Author:** ![Jose\_Angel\_Morena\_Si](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jose_angel_morena_si/32/76522_2.png) [@Jose\_Angel\_Morena\_Si](https://discuss.elastic.co/u/Jose_Angel_Morena_Si)\
**Post date:** [February 11, 2021, 2:30pm UTC](https://discuss.elastic.co/t/unable-to-setup-elasticsearch-cluster-mode/263987/6 "2021-02-11T14:30:17Z")

</div>

Hello, I got it working by bootstrapping the nodes one by one:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/7/979c9ae6337a8dc6595196d161c87da432169631.png)

There is this article that enlighted me to do so:

> **[Add and remove nodes in your cluster | Elasticsearch Reference \[7.x\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/7.x/add-elasticsearch-nodes.html)**

I removed the option `#discovery.zen.minimum_master_nodes` as it seems to be deprecated.

And had to remove the directory /var/lib/elasticsearch/nodes, node by node.

Hope this helps to anyone running into this issue.

Thanks for your help @ylasri

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 11, 2021, 2:30pm UTC](https://discuss.elastic.co/t/unable-to-setup-elasticsearch-cluster-mode/263987/7 "2021-03-11T14:30:22Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.