# Help forming a cluster

**URL:** <https://discuss.elastic.co/t/help-forming-a-cluster/235172>\
**Category:** Elasticsearch\
**Created:** [June 1, 2020, 1:11pm UTC](https://discuss.elastic.co/t/help-forming-a-cluster/235172 "2020-06-01T13:11:23Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kevin\_M](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kevin_m/32/69498_2.png) [@Kevin\_M](https://discuss.elastic.co/u/Kevin_M)\
**Post date:** [June 1, 2020, 1:11pm UTC](https://discuss.elastic.co/t/help-forming-a-cluster/235172/1 "2020-06-01T13:11:24Z")

</div>

Hey I am loosing my mind over this, I have 3 VMs on the same ESXI host in the same subnet. All have firewalld disabled and SElinux disabled. I can telnet on port 9300 from all hosts to all hosts in all directions. The cluster just will not form

here is config for hosts:

```auto
> path.data: /var/lib/elasticsearch
> path.logs: /var/log/elasticsearch
> cluster.name: "wazuh"
> node.name: "st-wazuh-es01"
> #network.bind_host: 172.16.40.90
> network.host: 172.16.40.90
> network.publish_host: 172.16.40.90
> node.master: true
> node.data: true
> discovery.zen.ping.unicast.hosts: ["172.16.40.90", "172.16.40.91", "172.16.40.92"]
> discovery.zen.minimum_master_nodes: 2
> cluster.initial_master_nodes:
> - 172.16.40.90
> - 172.16.40.91
> - 172.16.40.92
> 
> path.data: /var/lib/elasticsearch
> path.logs: /var/log/elasticsearch
> cluster.name: "wazuh"
> node.name: "st-wazuh-es02"
> network.bind_host: 172.16.40.91
> network.host: 172.16.40.91
> network.publish_host: 172.16.40.91
> node.master: true
> node.data: true
> discovery.zen.ping.unicast.hosts: ["172.16.40.90", "172.16.40.91", "172.16.40.92"]
> discovery.zen.minimum_master_nodes: 2
> cluster.initial_master_nodes:
> - 172.16.40.90
> - 172.16.40.91
> - 172.16.40.92
> 
> 
> path.data: /var/lib/elasticsearch
> path.logs: /var/log/elasticsearch
> cluster.name: "wazuh"
> node.name: "st-wazuh-es03"
> network.bind_host: 172.16.40.92
> network.host: 172.16.40.92
> network.publish_host: 172.16.40.92
> node.master: true
> node.data: true
> discovery.zen.ping.unicast.hosts: ["172.16.40.90", "172.16.40.91", "172.16.40.92"]
> discovery.zen.minimum_master_nodes: 2
> cluster.initial_master_nodes:
> - 172.16.40.90
> - 172.16.40.91
> - 172.16.40.92

```

when I cat /var/log/elasticsearch/elasticsearch.log - nothing showing for today even though I am bouncing the service

Here is output of every host:

```auto
> curl -XGET '172.16.40.92:9200/_cluster/health?pretty'
> {
> "cluster_name" : "wazuh",
> "status" : "yellow",
> "timed_out" : false,
> "number_of_nodes" : 1,
> "number_of_data_nodes" : 1,
> "active_primary_shards" : 50,
> "active_shards" : 50,
> "relocating_shards" : 0,
> "initializing_shards" : 0,
> "unassigned_shards" : 1,
> "delayed_unassigned_shards" : 0,
> "number_of_pending_tasks" : 0,
> "number_of_in_flight_fetch" : 0,
> "task_max_waiting_in_queue_millis" : 0,
> "active_shards_percent_as_number" : 98.0392156862745
> }
> 
> curl -XGET '172.16.40.90:9200/_cluster/health?pretty'
> {
> "cluster_name" : "wazuh",
> "status" : "yellow",
> "timed_out" : false,
> "number_of_nodes" : 1,
> "number_of_data_nodes" : 1,
> "active_primary_shards" : 58,
> "active_shards" : 58,
> "relocating_shards" : 0,
> "initializing_shards" : 0,
> "unassigned_shards" : 1,
> "delayed_unassigned_shards" : 0,
> "number_of_pending_tasks" : 0,
> "number_of_in_flight_fetch" : 0,
> "task_max_waiting_in_queue_millis" : 0,
> "active_shards_percent_as_number" : 98.30508474576271
> }
> 
> curl -XGET '172.16.40.91:9200/_cluster/health?pretty' {
> "cluster_name" : "wazuh",
> "status" : "yellow",
> "timed_out" : false,
> "number_of_nodes" : 1,
> "number_of_data_nodes" : 1,
> "active_primary_shards" : 50,
> "active_shards" : 50,
> "relocating_shards" : 0,
> "initializing_shards" : 0,
> "unassigned_shards" : 1,
> "delayed_unassigned_shards" : 0,
> "number_of_pending_tasks" : 0,
> "number_of_in_flight_fetch" : 0,
> "task_max_waiting_in_queue_millis" : 0,
> "active_shards_percent_as_number" : 98.0392156862745
> }

```

and versions

```auto
> {
> "name" : "st-wazuh-es01",
> "cluster_name" : "wazuh",
> "cluster_uuid" : "x9P_mXJ2Slu-aKBsYszGmA",
> "version" : {
> "number" : "7.6.2",
> "build_flavor" : "default",
> "build_type" : "rpm",
> "build_hash" : "ef48eb35cf30adf4db14086e8aabd07ef6fb113f",
> "build_date" : "2020-03-26T06:34:37.794943Z",
> "build_snapshot" : false,
> "lucene_version" : "8.4.0",
> "minimum_wire_compatibility_version" : "6.8.0",
> "minimum_index_compatibility_version" : "6.0.0-beta1"
> },
> "tagline" : "You Know, for Search"
> }
> 
> curl -XGET 'http://172.16.40.91:9200'
> {
> "name" : "st-wazuh-es02",
> "cluster_name" : "wazuh",
> "cluster_uuid" : "x9P_mXJ2Slu-aKBsYszGmA",
> "version" : {
> "number" : "7.6.2",
> "build_flavor" : "default",
> "build_type" : "rpm",
> "build_hash" : "ef48eb35cf30adf4db14086e8aabd07ef6fb113f",
> "build_date" : "2020-03-26T06:34:37.794943Z",
> "build_snapshot" : false,
> "lucene_version" : "8.4.0",
> "minimum_wire_compatibility_version" : "6.8.0",
> "minimum_index_compatibility_version" : "6.0.0-beta1"
> },
> "tagline" : "You Know, for Search"
> }
> 
> curl -XGET 'http://172.16.40.92:9200'
> {
> "name" : "st-wazuh-es03",
> "cluster_name" : "wazuh",
> "cluster_uuid" : "x9P_mXJ2Slu-aKBsYszGmA",
> "version" : {
> "number" : "7.6.2",
> "build_flavor" : "default",
> "build_type" : "rpm",
> "build_hash" : "ef48eb35cf30adf4db14086e8aabd07ef6fb113f",
> "build_date" : "2020-03-26T06:34:37.794943Z",
> "build_snapshot" : false,
> "lucene_version" : "8.4.0",
> "minimum_wire_compatibility_version" : "6.8.0",
> "minimum_index_compatibility_version" : "6.0.0-beta1"
> },
> "tagline" : "You Know, for Search"
> }

```

the one strange this is I cannot run curl against localhost - I have to use the IP address

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [June 1, 2020, 2:16pm UTC](https://discuss.elastic.co/t/help-forming-a-cluster/235172/2 "2020-06-01T14:16:16Z")

</div>

It looks like you have formed three distinct one-node clusters, although they all have the same cluster ID which indicates that you copied the data directory. Don't do that, each node should start with an empty data directory.

I think the simplest fix is to wipe all their data directories and start again.

Also `cluster.initial_master_nodes` should be set to the node names, not their IP addresses.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [June 1, 2020, 2:26pm UTC](https://discuss.elastic.co/t/help-forming-a-cluster/235172/4 "2020-06-01T14:26:38Z")

</div>

Some other comments on config:

> [@Kevin\_M](#):
>
> ```auto
> > network.bind_host: 172.16.40.92
> > network.host: 172.16.40.92
> > network.publish_host: 172.16.40.92
> 
> ```

This is redundant, you should remove `network.bind_host` and `network.publish_host` and only set `network.host`.

> [@Kevin\_M](#):
>
> ```auto
> > discovery.zen.ping.unicast.hosts: ["172.16.40.90", "172.16.40.91", "172.16.40.92"]
> 
> ```

This is deprecated, you should set `discovery.seed_hosts` instead.

> [@Kevin\_M](#):
>
> ```auto
> > discovery.zen.minimum_master_nodes: 2
> 
> ```

This is deprecated and does nothing, you should remove this line.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [June 1, 2020, 2:28pm UTC](https://discuss.elastic.co/t/help-forming-a-cluster/235172/5 "2020-06-01T14:28:37Z")

</div>

One more thing 😁

> [@Kevin\_M](#):
>
> when I cat /var/log/elasticsearch/elasticsearch.log - nothing showing for today even though I am bouncing the service

Yes, the log file is named after the cluster, so I think you want to look at `/var/log/elasticsearch/wazuh.log`.

---

<div class="post-metadata">

**Author:** ![Kevin\_M](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kevin_m/32/69498_2.png) [@Kevin\_M](https://discuss.elastic.co/u/Kevin_M)\
**Post date:** [June 1, 2020, 2:32pm UTC](https://discuss.elastic.co/t/help-forming-a-cluster/235172/6 "2020-06-01T14:32:37Z")

</div>

> [@DavidTurner](#):
>
> g, you should remove this line.

thanks David,  
Is seed host supposed to use IP or hostname

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [June 1, 2020, 2:37pm UTC](https://discuss.elastic.co/t/help-forming-a-cluster/235172/7 "2020-06-01T14:37:55Z")

</div>

[Yes, that's right](https://www.elastic.co/guide/en/elasticsearch/reference/current/discovery-settings.html#unicast.hosts):

> Each address can be either an IP address or a hostname which resolves to one or more IP addresses via DNS.

---

<div class="post-metadata">

**Author:** ![Kevin\_M](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kevin_m/32/69498_2.png) [@Kevin\_M](https://discuss.elastic.co/u/Kevin_M)\
**Post date:** [June 1, 2020, 2:43pm UTC](https://discuss.elastic.co/t/help-forming-a-cluster/235172/8 "2020-06-01T14:43:31Z")

</div>

> [@DavidTurner](#):
>
> /var/log/elasticsearch/wazuh.log

OK thank you so much!!  
I rm -rf \* the /var/lib/elasticsearch/node folder and #2 and #3 are now in a cluster but #1 is not yet - still in his own little island  
I also updated the config:

```auto
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
cluster.name: "wazuh"
node.name: "st-wazuh-es02"
network.host: 0.0.0.0
node.master: true
node.data: true
discovery.seed_hosts:
 - 172.16.40.90
 - 172.16.40.91
 - 172.16.40.92
cluster.initial_master_nodes:
 - st-wazuh-es01
 - st-wazuh-es02
 - st-wazuh-es03

```

I will say they (#1) and the rest have different cluster UUID

---

<div class="post-metadata">

**Author:** ![Kevin\_M](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kevin_m/32/69498_2.png) [@Kevin\_M](https://discuss.elastic.co/u/Kevin_M)\
**Post date:** [June 1, 2020, 3:06pm UTC](https://discuss.elastic.co/t/help-forming-a-cluster/235172/9 "2020-06-01T15:06:24Z")

</div>

also found this thread from you also 🙂

> [@Change ES cluster\_uuid](https://discuss.elastic.co/t/change-es-cluster-uuid/193092/2):
>
> That would explain it slight_smile It's best not to clone nodes, it can cause all sorts of strange issues. Yes, [the elasticsearch-node unsafe-bootstrap tool](https://www.elastic.co/guide/en/elasticsearch/reference/current/node-tool.html) will generate a new cluster UUID.

I did initially only have host#1 - then VM cloned it twice.

Deleting the data file for #1 fixed the issue - hoping i dont have any more issues down the road due to cloning

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 29, 2020, 3:14pm UTC](https://discuss.elastic.co/t/help-forming-a-cluster/235172/10 "2020-06-29T15:14:41Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.