# Nodes unable to join cluster

**URL:** <https://discuss.elastic.co/t/nodes-unable-to-join-cluster/192823>\
**Category:** Elasticsearch\
**Created:** [July 30, 2019, 5:21am UTC](https://discuss.elastic.co/t/nodes-unable-to-join-cluster/192823 "2019-07-30T05:21:09Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![msk\_76](https://avatars.discourse-cdn.com/v4/letter/m/dbc845/32.png) [@msk\_76](https://discuss.elastic.co/u/msk_76)\
**Post date:** [July 30, 2019, 5:21am UTC](https://discuss.elastic.co/t/nodes-unable-to-join-cluster/192823/1 "2019-07-30T05:21:09Z")

</div>

I have following node configurations for 3 node cluster :  
Each node has two network cards, one for private IP for internode communication and another for public IP to connect from kibana/ external users.

The private IP's can't be reachable from outside than of 3 nodes.

**Node-1 :**  
cluster.name: cad-elastisearch  
node.name: node-1-gnbsx00043  
path.data: /data/esdb/data  
path.logs: /data/esdb/log  
network.host: 10.129.212.43 \<= public IP  
http.host: 10.129.212.43  
transport.host: 10.129.213.43 \<= private IP  
http.port: 9200  
discovery.zen.ping.unicast.hosts: ["10.129.212.43","10.129.212.44","10.129.212.45"] \<= Public IP's  
discovery.zen.minimum\_master\_nodes: 2

**Node-2**  
cluster.name: cad-elastisearch  
node.name: node-2-gnbsx00044  
path.data: /data/esdb/data  
path.logs: /data/esdb/log  
network.host: 10.129.212.44  
http.host: 10.129.212.44  
transport.host: 10.129.213.44  
http.port: 9200  
discovery.zen.ping.unicast.hosts: ["10.129.212.43","10.129.212.44","10.129.212.45"]  
discovery.zen.minimum\_master\_nodes: 2

**Node-3**  
cluster.name: cad-elastisearch  
node.name: node-2-gnbsx00044  
path.data: /data/esdb/data  
path.logs: /data/esdb/log  
network.host: 10.129.212.44  
http.host: 10.129.212.44  
transport.host: 10.129.213.44  
http.port: 9200  
discovery.zen.ping.unicast.hosts: ["10.129.212.43","10.129.212.44","10.129.212.45"]  
discovery.zen.minimum\_master\_nodes: 2

**With this configuration the error log shows**

_[2019-07-30T06:58:56,639][INFO][o.e.p.PluginsService] [node-1-gnbsx00043] loaded module [x-pack-monitoring]_  
_[2019-07-30T06:58:56,640][INFO][o.e.p.PluginsService] [node-1-gnbsx00043] loaded module [x-pack-rollup]_  
_[2019-07-30T06:58:56,640][INFO][o.e.p.PluginsService] [node-1-gnbsx00043] loaded module [x-pack-security]_  
_[2019-07-30T06:58:56,640][INFO][o.e.p.PluginsService] [node-1-gnbsx00043] loaded module [x-pack-sql]_  
_[2019-07-30T06:58:56,640][INFO][o.e.p.PluginsService] [node-1-gnbsx00043] loaded module [x-pack-upgrade]_  
_[2019-07-30T06:58:56,640][INFO][o.e.p.PluginsService] [node-1-gnbsx00043] loaded module [x-pack-watcher]_  
_[2019-07-30T06:58:56,640][INFO][o.e.p.PluginsService] [node-1-gnbsx00043] no plugins loaded_  
_[2019-07-30T06:59:00,292][INFO][o.e.x.s.a.s.FileRolesStore] [node-1-gnbsx00043] parsed [0] roles from file [/data/essw/elasticsearch-6.6.0/config/roles.yml]_  
_[2019-07-30T06:59:00,796][INFO][o.e.x.m.p.l.CppLogMessageHandler] [node-1-gnbsx00043] [controller/237437] [Main.cc@109] controller (64 bit): Version 6.6.0 (Build bbb4919f4d17a5) Copyright (c) 2019 Elasticsearch BV_  
_[2019-07-30T06:59:01,187][DEBUG][o.e.a.ActionModule] [node-1-gnbsx00043] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security_  
_[2019-07-30T06:59:01,400][INFO][o.e.d.DiscoveryModule] [node-1-gnbsx00043] using discovery type [zen] and host providers [settings]_  
_[2019-07-30T06:59:02,099][INFO][o.e.n.Node] [node-1-gnbsx00043] initialized_  
_[2019-07-30T06:59:02,099][INFO][o.e.n.Node] [node-1-gnbsx00043] starting ..._  
_[2019-07-30T06:59:02,237][INFO][o.e.t.TransportService] [node-1-gnbsx00043] publish\_address {10.129.213.43:9300}, bound\_addresses {10.129.213.43:9300}_  
_[2019-07-30T06:59:02,255][INFO][o.e.b.BootstrapChecks] [node-1-gnbsx00043] bound or publishing to a non-loopback address, enforcing bootstrap checks_  
_[2019-07-30T06:59:05,308][WARN][o.e.d.z.ZenDiscovery] [node-1-gnbsx00043] not enough master nodes discovered during pinging (found [[Candidate{node={node-1-gnbsx00043}{78KaKTlkSJefU6tt3ZRPkg}{7m9ar3gEQ1OIIq1u-pFOXQ}{10.129.213.43}{10.129.213.43:9300}{ml.machine\_memory=270841049088, xpack.installed=true, ml.max\_open\_jobs=20, ml.enabled=true}, clusterStateVersion=-1}]], but needed [2]), pinging again_  
_[2019-07-30T06:59:08,309][WARN][o.e.d.z.ZenDiscovery] [node-1-gnbsx00043] not enough master nodes discovered during pinging (found [[Candidate{node={node-1-gnbsx00043}{78KaKTlkSJefU6tt3ZRPkg}{7m9ar3gEQ1OIIq1u-pFOXQ}{10.129.213.43}{10.129.213.43:9300}{ml.machine\_memory=270841049088, xpack.installed=true, ml.max\_open\_jobs=20, ml.enabled=true}, clusterStateVersion=-1}]], but needed [2]), pinging again_  
_[2019-07-30T06:59:11,310][WARN][o.e.d.z.ZenDiscovery] [node-1-gnbsx00043] not enough master nodes discovered during pinging (found [[Candidate{node={node-1-gnbsx00043}{78KaKTlkSJefU6tt3ZRPkg}{7m9ar3gEQ1OIIq1u-pFOXQ}{10.129.213.43}{10.129.213.43:9300}{ml.machine\_memory=270841049088, xpack.installed=true, ml.max\_open\_jobs=20, ml.enabled=true}, clusterStateVersion=-1}]], but needed [2]), pinging again_  
_[2019-07-30T06:59:14,311][WARN][o.e.d.z.ZenDiscovery] [node-1-gnbsx00043] not enough master nodes discovered during pinging (found [[Candidate{node={node-1-gnbsx00043}{78KaKTlkSJefU6tt3ZRPkg}{7m9ar3gEQ1OIIq1u-pFOXQ}{10.129.213.43}{10.129.213.43:9300}{ml.machine\_memory=270841049088, xpack.installed=true, ml.max\_open\_jobs=20, ml.enabled=true}, clusterStateVersion=-1}]], but needed [2]), pinging again_

---

<div class="post-metadata">

**Author:** ![Bernt\_Rostad](https://avatars.discourse-cdn.com/v4/letter/b/3ab097/32.png) [@Bernt\_Rostad](https://discuss.elastic.co/u/Bernt_Rostad)\
**Post date:** [July 30, 2019, 7:40am UTC](https://discuss.elastic.co/t/nodes-unable-to-join-cluster/192823/2 "2019-07-30T07:40:15Z")

</div>

It seems you've given Node-2 and Node-3 the same name and configuration:

> [@msk\_76](#):
>
> **Node-2**  
> node.name: node-2-gnbsx00044  
> http.host: 10.129.212.44  
> http.port: 9200

> [@msk\_76](#):
>
> **Node-3**  
> node.name: node-2-gnbsx00044  
> http.host: 10.129.212.44  
> http.port: 9200

You will have to give Node-3 it's own node.name and host-information to separate it from Node-2 and allow both to join the cluster.

---

<div class="post-metadata">

**Author:** ![msk\_76](https://avatars.discourse-cdn.com/v4/letter/m/dbc845/32.png) [@msk\_76](https://discuss.elastic.co/u/msk_76)\
**Post date:** [July 30, 2019, 8:18am UTC](https://discuss.elastic.co/t/nodes-unable-to-join-cluster/192823/3 "2019-07-30T08:18:01Z")

</div>

Sorry it was a typo during pasting in the discussion forum.

In the node configuration of node-3 it is

cluster.name: cad-elastisearch  
node.name: node-1-gnbsx00044  
path.data: /data/esdb/data  
path.logs: /data/esdb/log  
network.host: 10.129.212.45  
http.host: 10.129.212.45  
transport.host: 10.129.213.45  
http.port: 9200  
discovery.zen.ping.unicast.hosts: ["10.129.212.43","10.129.212.44","10.129.212.45"]  
discovery.zen.minimum\_master\_nodes: 2

---

<div class="post-metadata">

**Author:** ![msk\_76](https://avatars.discourse-cdn.com/v4/letter/m/dbc845/32.png) [@msk\_76](https://discuss.elastic.co/u/msk_76)\
**Post date:** [July 30, 2019, 8:20am UTC](https://discuss.elastic.co/t/nodes-unable-to-join-cluster/192823/4 "2019-07-30T08:20:08Z")

</div>

Again a type. Please discard the previous.

In the node configuration of node-3, it is

cluster.name: cad-elastisearch  
node.name: node-1-gnbsx00045  
path.data: /data/esdb/data  
path.logs: /data/esdb/log  
network.host: 10.129.212.45  
http.host: 10.129.212.45  
transport.host: 10.129.213.45  
http.port: 9200  
discovery.zen.ping.unicast.hosts: ["10.129.212.43","10.129.212.44","10.129.212.45"]  
discovery.zen.minimum\_master\_nodes: 2

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [July 30, 2019, 9:02am UTC](https://discuss.elastic.co/t/nodes-unable-to-join-cluster/192823/5 "2019-07-30T09:02:39Z")

</div>

Unicast hosts should as far as I can tell use the private IP addresses.

---

<div class="post-metadata">

**Author:** ![msk\_76](https://avatars.discourse-cdn.com/v4/letter/m/dbc845/32.png) [@msk\_76](https://discuss.elastic.co/u/msk_76)\
**Post date:** [July 30, 2019, 9:35am UTC](https://discuss.elastic.co/t/nodes-unable-to-join-cluster/192823/6 "2019-07-30T09:35:31Z")

</div>

Initially I tried using private IP list in unicast but the error was same. These nodes were unable to recognize each other.

---

<div class="post-metadata">

**Author:** ![msk\_76](https://avatars.discourse-cdn.com/v4/letter/m/dbc845/32.png) [@msk\_76](https://discuss.elastic.co/u/msk_76)\
**Post date:** [July 30, 2019, 9:40am UTC](https://discuss.elastic.co/t/nodes-unable-to-join-cluster/192823/7 "2019-07-30T09:40:02Z")

</div>

Here is output of cluster health :

{"cluster\_name":"lsf-elasticsearch","status":"yellow","timed\_out":false,"number\_of\_nodes":1,"number\_of\_data\_nodes":1,"active\_primary\_shards":182,"active\_shards":182,"relocating\_shards":0,"initializing\_shards":0,"unassigned\_shards":181,"delayed\_unassigned\_shards":0,"number\_of\_pending\_tasks":0,"number\_of\_in\_flight\_fetch":0,"task\_max\_waiting\_in\_queue\_millis":0,"active\_shards\_percent\_as\_number":50.13774104683195}

---

<div class="post-metadata">

**Author:** ![msk\_76](https://avatars.discourse-cdn.com/v4/letter/m/dbc845/32.png) [@msk\_76](https://discuss.elastic.co/u/msk_76)\
**Post date:** [July 30, 2019, 9:42am UTC](https://discuss.elastic.co/t/nodes-unable-to-join-cluster/192823/8 "2019-07-30T09:42:21Z")

</div>

Pardon for my keyboard problem. The last cluster health was wrongly pasted.  
This is the current state :  
{"error":{"root\_cause":[{"type":"master\_not\_discovered\_exception","reason":null}],"type":"master\_not\_discovered\_exception","reason":null},"status":503}

---

<div class="post-metadata">

**Author:** ![Bernt\_Rostad](https://avatars.discourse-cdn.com/v4/letter/b/3ab097/32.png) [@Bernt\_Rostad](https://discuss.elastic.co/u/Bernt_Rostad)\
**Post date:** [July 30, 2019, 9:59am UTC](https://discuss.elastic.co/t/nodes-unable-to-join-cluster/192823/9 "2019-07-30T09:59:08Z")

</div>

> [@msk\_76](#):
>
> transport.host: 10.129.213.45  
> http.port: 9200  
> discovery.zen.ping.unicast.hosts: ["10.129.212.43","10.129.212.44","10.129.212.45"]

I normally don't specify the transport.host in my elasticsearch.yml files, so I missed that part in your settings.

When specifying **transport.host** , you must use those values, not the http.host, in your **discovery.zen.ping.unicast.hosts** configuration because Elasticsearch uses TCP for internal master / node discovery and HTTP only for client communication (searches etc).

To solve the issue you should either remove the transport.host settings, which will cause Elasticsearch to use the http.host instead, or configure the Zen discovery something like this:

`discovery.zen.ping.unicast.hosts: ["10.129.213.43","10.129.213.44","10.129.213.45"]`

where I've replaced the http hosts 10.129. **212**.4x with 10.129. **213**.4x. Either way should work.

Good luck!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 27, 2019, 9:59am UTC](https://discuss.elastic.co/t/nodes-unable-to-join-cluster/192823/10 "2019-08-27T09:59:23Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.