# Unable to join two nodes in cluster **URL:** \ **Category:** Elasticsearch\ **Created:** [July 19, 2019, 12:17pm UTC](https://discuss.elastic.co/t/unable-to-join-two-nodes-in-cluster/191390 "2019-07-19T12:17:46Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![enVolt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/envolt/32/43550_2.png) [@enVolt](https://discuss.elastic.co/u/enVolt)\ **Post date:** [July 19, 2019, 12:17pm UTC](https://discuss.elastic.co/t/unable-to-join-two-nodes-in-cluster/191390/1 "2019-07-19T12:17:47Z")
Hello, I'm trying to set up two ES node in cluster nodes, and has absolutely given up as I'm posting this. I've two instance running, with following configs - ```auto # es-01 cluster.name: dc-world node.name: smallville path.data: /es/data path.logs: /es/log network.host: 0.0.0.0 discovery.seed_hosts: ["10.116.48.116"] # IP of es-02 cluster.initial_master_nodes: ["gotham", "smallville"] ``` ```auto # es-02 cluster.name: dc-world node.name: gotham path.data: /es/data path.logs: /es/log network.host: 0.0.0.0 network.publish_host: _site_ cluster.initial_master_nodes: ["gotham", "smallville"] ``` I'm getting following error (from es-01 logs) (stack trace reduced; due to post limit) - ```auto [2019-07-19T06:33:02,824][WARN][o.e.c.c.ClusterFormationFailureHelper] [smallville] master not discovered or elected yet, an election requires two nodes with ids [JYKLqIsvR0yruH0ecPG4wA, CTAdAiYbT-ajAXtSVEs3Bw], have discovered [{gotham}{CTAdAiYbT-ajAXtSVEs3Bw}{V18vdJW3TyKlfMJIJhlGkQ}{10.116.48.116}{10.116.48.116:9300}{ml.machine_memory=16651354112, ml.max_open_jobs=20, xpack.installed=true}] which is not a quorum; discovery will continue using [10.116.48.116:9300] from hosts providers and [{smallville}{JYKLqIsvR0yruH0ecPG4wA}{-9WFnGHqQnaxid1cV4oAHg}{161.202.2.237}{161.202.2.237:9300}{ml.machine_memory=16651354112, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 149, last-accepted version 0 in term 0 [2019-07-19T06:33:06,904][INFO][o.e.c.c.JoinHelper] [smallville] failed to join {smallville}{JYKLqIsvR0yruH0ecPG4wA}{-9WFnGHqQnaxid1cV4oAHg}{161.202.2.237}{161.202.2.237:9300}{ml.machine_memory=16651354112, xpack.installed=true, ml.max_open_jobs=20} with JoinRequest{sourceNode={smallville}{JYKLqIsvR0yruH0ecPG4wA}{-9WFnGHqQnaxid1cV4oAHg}{161.202.2.237}{161.202.2.237:9300}{ml.machine_memory=16651354112, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=150, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={smallville}{JYKLqIsvR0yruH0ecPG4wA}{-9WFnGHqQnaxid1cV4oAHg}{161.202.2.237}{161.202.2.237:9300}{ml.machine_memory=16651354112, xpack.installed=true, ml.max_open_jobs=20}, targetNode={smallville}{JYKLqIsvR0yruH0ecPG4wA}{-9WFnGHqQnaxid1cV4oAHg}{161.202.2.237}{161.202.2.237:9300}{ml.machine_memory=16651354112, xpack.installed=true, ml.max_open_jobs=20}}]} org.elasticsearch.transport.RemoteTransportException: [smallville][161.202.2.237:9300][internal:cluster/coordination/join] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: received a newer join from {smallville}{JYKLqIsvR0yruH0ecPG4wA}{-9WFnGHqQnaxid1cV4oAHg}{161.202.2.237}{161.202.2.237:9300}{ml.machine_memory=16651354112, xpack.installed=true, ml.max_open_jobs=20} at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.handleJoinRequest(JoinHelper.java:451) [elasticsearch-7.2.0.jar:7.2.0] ...... .... ``` es-02 logs - ```auto [2019-07-19T06:37:45,288][WARN][o.e.c.c.ClusterFormationFailureHelper] [gotham] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [gotham, smallville] to bootstrap a cluster: have discovered []; discovery will continue using [127.0.0.1:9300, 127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304, 127.0.0.1:9305, [::1]:9300, [::1]:9301, [::1]:9302, [::1]:9303, [::1]:9304, [::1]:9305] from hosts providers and [{gotham}{CTAdAiYbT-ajAXtSVEs3Bw}{V18vdJW3TyKlfMJIJhlGkQ}{10.116.48.116}{10.116.48.116:9300}{ml.machine_memory=16651354112, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 218, last-accepted version 0 in term 0 [2019-07-19T06:38:28,567][INFO][o.e.c.c.JoinHelper] [gotham] failed to join {smallville}{JYKLqIsvR0yruH0ecPG4wA}{-9WFnGHqQnaxid1cV4oAHg}{161.202.2.237}{161.202.2.237:9300}{ml.machine_memory=16651354112, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={gotham}{CTAdAiYbT-ajAXtSVEs3Bw}{V18vdJW3TyKlfMJIJhlGkQ}{10.116.48.116}{10.116.48.116:9300}{ml.machine_memory=16651354112, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=225, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={gotham}{CTAdAiYbT-ajAXtSVEs3Bw}{V18vdJW3TyKlfMJIJhlGkQ}{10.116.48.116}{10.116.48.116:9300}{ml.machine_memory=16651354112, xpack.installed=true, ml.max_open_jobs=20}, targetNode={smallville}{JYKLqIsvR0yruH0ecPG4wA}{-9WFnGHqQnaxid1cV4oAHg}{161.202.2.237}{161.202.2.237:9300}{ml.machine_memory=16651354112, ml.max_open_jobs=20, xpack.installed=true}}]} org.elasticsearch.transport.NodeNotConnectedException: [smallville][161.202.2.237:9300] Node not connected at org.elasticsearch.transport.ConnectionManager.getConnection(ConnectionManager.java:151) ~[elasticsearch-7.2.0.jar:7.2.0] ... ... ``` This has been after numerous failed attempt, and after reading/digesting everything I could find (forming single node clusters; security group rule issues; understanding 9200 vs 9300; almost failing to understand unicast; reading articles about 6.8's zen discovery) Before I added `network.publish_host: _site_` in es-02 config, I was getting this in the logs (from es-01; similar on es-02) - ```auto [2019-07-19T05:53:55,034][WARN][o.e.c.c.ClusterFormationFailureHelper] [smallville] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [10.116.48.112, 10.116.48.116] to bootstrap a cluster: have discovered []; discovery will continue using [127.0.0.1:9300, 127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304, 127.0.0.1:9305, [::1]:9300, [::1]:9301, [::1]:9302, [::1]:9303, [::1]:9304, [::1]:9305] from hosts providers and [{smallville}{8MEe4svaTseS-P3BwzjK9A}{ecMet_NNRv2arG-ZAuuq7g}{161.202.2.237}{161.202.2.237:9300}{ml.machine_memory=16651354112, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0 ``` If you need any more info; please let me know. I'm on version 7.2.0 (build 508c38a from rpm). OS - CentOS 7 ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 19, 2019, 1:36pm UTC](https://discuss.elastic.co/t/unable-to-join-two-nodes-in-cluster/191390/2 "2019-07-19T13:36:24Z")
I am guessing you've tried a bunch of different settings and it looks like the nodes have got into a bit of a funny state as a result. Has this cluster ever worked? If not, and it contains no data, then the best thing to try is: - shut both of the nodes down - delete the contents of `/es/data` on both nodes - start them up again The configs look reasonable, but perhaps are not wholly consistent with the state on disk any more, so this'll reset that state and give you a clean start. One other thing: it's generally best to put the addresses of _all_ master-eligible nodes in the `discovery.seed_hosts` setting on each node. Even the local node. Not essential, but if your nodes are more symmetric then future troubleshooting will likely be easier. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 19, 2019, 1:45pm UTC](https://discuss.elastic.co/t/unable-to-join-two-nodes-in-cluster/191390/3 "2019-07-19T13:45:28Z")
Hmm actually it looks like you also have connectivity issues. `gotham` seems to think its address is `10.116.48.116:9300` but `smallville` is `161.202.2.237:9300`. Should they really have such wildly different addresses? Why do you have `network.publish_host: _site_` on one and not the other? Can you try getting rid of this and setting `network.host` to each node's proper address instead of `0.0.0.0`? ---
**Author:** ![enVolt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/envolt/32/43550_2.png) [@enVolt](https://discuss.elastic.co/u/enVolt)\ **Post date:** [July 22, 2019, 6:20am UTC](https://discuss.elastic.co/t/unable-to-join-two-nodes-in-cluster/191390/4 "2019-07-22T06:20:08Z")
`161.202.2.237:9300` is the public IP. Both the instances are inside VPC, hence I tried changing the `publish_host` address to local IP (but only on a single node; i assumed it to be just for discovery purpose; if A can reach B, B can communicate with A and find out where is A) I missed the fact that `network.host` is also for identifying host. When I put `0.0.0.0`, I wanted it to be accessible from anywhere (I had Security Groups in place for disabling public access) Just adding following two lines did the charm - ```auto network.host: _site_ network.publish_host: _site_ ``` (Also I cleaned both the data and log directory before changing the config) Though, the bind address is now private network IP, which is absolutely fine. Thanks, ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 22, 2019, 6:32am UTC](https://discuss.elastic.co/t/unable-to-join-two-nodes-in-cluster/191390/5 "2019-07-22T06:32:45Z")
> [@enVolt](#): > > ```auto > network.publish_host: _site_ > > ``` This setting defaults to being equal to `network.host` so I think you do not need to set it explicitly like this. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 19, 2019, 6:32am UTC](https://discuss.elastic.co/t/unable-to-join-two-nodes-in-cluster/191390/6 "2019-08-19T06:32:46Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.