# Node does not join the cluster **URL:** \ **Category:** Elasticsearch\ **Created:** [October 18, 2012, 7:17pm UTC](https://discuss.elastic.co/t/node-does-not-join-the-cluster/9415 "2012-10-18T19:17:19Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Michael\_Smolyak](https://avatars.discourse-cdn.com/v4/letter/m/7cd45c/32.png) [@Michael\_Smolyak](https://discuss.elastic.co/u/Michael_Smolyak)\ **Post date:** [October 18, 2012, 7:17pm UTC](https://discuss.elastic.co/t/node-does-not-join-the-cluster/9415/1 "2012-10-18T19:17:19Z")
I am trying to implement failover in cluster with two nodes. When I test the configuration of two nodes (two tomcat servers, each running a Web app that starts an embedded ES server) running on the same machine, the reliably work. The two nodes cluster together and the cluster status is green. { - cluster\_name: "elasticsearch", - status: "green", - timed\_out: false, - number\_of\_nodes: 2, - number\_of\_data\_nodes: 2, - active\_primary\_shards: 5, - active\_shards: 10, - relocating\_shards: 0, - initializing\_shards: 0, - unassigned\_shards: 0 } Starting the same two servers on two different machines sometimes results in a green cluster, but more often than not, the two start as two separate instances, both with status yellow. During a successful startup attempt when the second server of the cluster starts up there is a lengthy handshake (I enabled Java debug-level logging for package org.elasticseach): 2012-10-18 14:37:15,200 EDT [elasticsearch[Killmonger, Erik][generic][T#9]] > DEBUG[org.elasticsearch.transport.netty]: [Killmonger, Erik] connected to > node [[Anomalito][XqHrYxdEQIagjLGVWokCwQ][inet[/192.168.35.137:9300]]] 2012-10-18 14:37:21,922 EDT [elasticsearch[Killmonger, > Erik][clusterService#updateTask][T#1]] > DEBUG[org.elasticsearch.cluster.service]: [Killmonger, Erik] processing > [zen-disco-receive(join from > node[[Anomalito][XqHrYxdEQIagjLGVWokCwQ][inet[/192.168.35.137:9300]]])]: > execute 2012-10-18 14:37:21,976 EDT [elasticsearch[Killmonger, > Erik][clusterService#updateTask][T#1]] > DEBUG[org.elasticsearch.cluster.service]: [Killmonger, Erik] cluster state > updated, version [8], source [zen-disco-receive(join from > node[[Anomalito][XqHrYxdEQIagjLGVWokCwQ][inet[/192.168.35.137:9300]]])] 2012-10-18 14:37:21,977 EDT [elasticsearch[Killmonger, > Erik][clusterService#updateTask][T#1]] INFO > [org.elasticsearch.cluster.service]: [Killmonger, Erik] added > {[Anomalito][XqHrYxdEQIagjLGVWokCwQ][inet[/192.168.35.137:9300]],}, reason: > zen-disco-receive(join from > node[[Anomalito][XqHrYxdEQIagjLGVWokCwQ][inet[/192.168.35.137:9300]]]) 2012-10-18 14:37:22,043 EDT [elasticsearch[Killmonger, > Erik][clusterService#updateTask][T#1]] > DEBUG[org.elasticsearch.cluster.service]: [Killmonger, Erik] processing > [zen-disco-receive(join from > node[[Anomalito][XqHrYxdEQIagjLGVWokCwQ][inet[/192.168.35.137:9300]]])]: > done applying updated cluster\_state 2012-10-18 14:37:22,043 EDT [elasticsearch[Killmonger, > Erik][riverClusterService#updateTask][T#1]] > DEBUG[org.elasticsearch.river.cluster]: [Killmonger, Erik] processing > [reroute\_rivers\_node\_changed]: execute 2012-10-18 14:37:22,044 EDT [elasticsearch[Killmonger, > Erik][riverClusterService#updateTask][T#1]] > DEBUG[org.elasticsearch.river.cluster]: [Killmonger, Erik] processing > [reroute\_rivers\_node\_changed]: no change in cluster\_state 2012-10-18 14:37:24,902 EDT [elasticsearch[Killmonger, > Erik][clusterService#updateTask][T#1]] > DEBUG[org.elasticsearch.cluster.service]: [Killmonger, Erik] processing > [routing-table-updater]: execute 2012-10-18 14:37:25,122 EDT [elasticsearch[Killmonger, > Erik][clusterService#updateTask][T#1]] > DEBUG[org.elasticsearch.cluster.service]: [Killmonger, Erik] cluster state > updated, version [9], source [routing-table-updater] 2012-10-18 14:37:25,150 EDT [elasticsearch[Killmonger, > Erik][riverClusterService#updateTask][T#1]] > DEBUG[org.elasticsearch.river.cluster]: [Killmonger, Erik] processing > [reroute\_rivers\_node\_changed]: execute 2012-10-18 14:37:25,150 EDT [elasticsearch[Killmonger, > Erik][riverClusterService#updateTask][T#1]] > DEBUG[org.elasticsearch.river.cluster]: [Killmonger, Erik] processing > [reroute\_rivers\_node\_changed]: no change in cluster\_state 2012-10-18 14:37:25,179 EDT [elasticsearch[Killmonger, > Erik][clusterService#updateTask][T#1]] > DEBUG[org.elasticsearch.cluster.service]: [Killmonger, Erik] processing > [routing-table-updater]: done applying updated cluster\_state 2012-10-18 14:37:32,667 EDT [elasticsearch[Killmonger, > Erik][clusterService#updateTask][T#1]] > DEBUG[org.elasticsearch.cluster.service]: [Killmonger, Erik] processing > [create-index [marketplace], cause [api]]: execute ... When I shut down one of the servers and then restart it, more often then not, the tomcat server starts up successfully, but the second ES server does not form the cluster with the running node. The two ES servers have yellow status (since replication factor is 1), but otherwise appear functional. No error messages are displayed. Instead of the lengthy stream of messages you see during a successful startup only a single message is emitted by the master: 2012-10-18 15:11:36,268 EDT [elasticsearch[Candra][generic][T#11]] > DEBUG[org.elasticsearch.transport.netty]: [Candra] connected to node > [[Jester][D9vXW5OrSUSrwuDaR0SIVA][inet[/192.168.35.137:9300]]] This message seems to imply that the two servers connected, but they do not forma cluster. Connecting to each server in the cluster gives me this status: { - cluster\_name: "elasticsearch", - status: "yellow", - timed\_out: false, - number\_of\_nodes: 1, - number\_of\_data\_nodes: 1, - active\_primary\_shards: 5, - active\_shards: 5, - relocating\_shards: 0, - initializing\_shards: 0, - unassigned\_shards: 5 } Both Candra and Jester run, but as two standalone ES servers. What could be the cause of this occasional failure to connect? How do I debug it? Thank you, Michael -- ---
**Author:** ![Michael\_Smolyak](https://avatars.discourse-cdn.com/v4/letter/m/7cd45c/32.png) [@Michael\_Smolyak](https://discuss.elastic.co/u/Michael_Smolyak)\ **Post date:** [October 18, 2012, 9:32pm UTC](https://discuss.elastic.co/t/node-does-not-join-the-cluster/9415/2 "2012-10-18T21:32:50Z")
I noticed that in case of faulty startup the lone message about server connection is followed by messages from Tomcat: 2012-10-18 17:32:30,813 EDT [elasticsearch[Berzerker][generic][T#7]] > DEBUG[org.elasticsearch.transport.netty]: [Berzerker] connected to node > [[Dreadknight][fqLtglKzTJ6uit2aYA\_YAw][inet[/192.168.35.137:9300]]] 2012-10-18 17:32:54,160 [Membership-MemberExpired.] INFO > org.apache.catalina.tribes.group.interceptors.TcpFailureDetector- Received > memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{127, > 0, 0, 1}:4001,{127, 0, 0, 1},4001, alive=95012, securePort=-1, UDP Port=-1, > id={-4 33 -44 -69 -27 -103 68 -21 -88 -63 -92 120 -64 -55 -100 -36 }, > payload={}, command={}, domain={}, ]] message. Will verify. 2012-10-18 17:32:54,162 [Membership-MemberExpired.] INFO > org.apache.catalina.tribes.group.interceptors.TcpFailureDetector- > Verification complete. Member still > alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{127, 0, 0, > 1}:4001,{127, 0, 0, 1},4001, alive=95012, securePort=-1, UDP Port=-1, > id={-4 33 -44 -69 -27 -103 68 -21 -88 -63 -92 120 -64 -55 -100 -36 }, > payload={}, command={}, domain={}, ]] 2012-10-18 17:32:57,675 [Membership-MemberExpired.] INFO > org.apache.catalina.tribes.group.interceptors.TcpFailureDetector- Received > memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{127, > 0, 0, 1}:4001,{127, 0, 0, 1},4001, alive=98395, securePort=-1, UDP Port=-1, > id={-4 33 -44 -69 -27 -103 68 -21 -88 -63 -92 120 -64 -55 -100 -36 }, > payload={}, command={}, domain={}, ]] message. Will verify. 2012-10-18 17:32:57,680 [Membership-MemberExpired.] INFO > org.apache.catalina.tribes.group.interceptors.TcpFailureDetector- > Verification complete. Member still > alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{127, 0, 0, > 1}:4001,{127, 0, 0, 1},4001, alive=98395, securePort=-1, UDP Port=-1, > id={-4 33 -44 -69 -27 -103 68 -21 -88 -63 -92 120 -64 -55 -100 -36 }, > payload={}, command={}, domain={}, ]] I wonder whether there is a relationship between memberDisappeared message from Tomcat and inability of Elasticsearch nodes to form a cluster. Michael -- ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [October 21, 2012, 1:16am UTC](https://discuss.elastic.co/t/node-does-not-join-the-cluster/9415/3 "2012-10-21T01:16:37Z")
I am not really sure, but you can turn on the discovery (or org.elasticsearch.discovery if running embedded) logging to see why it fails to discovery the other node. On Oct 18, 2012, at 11:32 PM, Michael Smolyak [msmolyak.nextcentury@gmail.com](mailto:msmolyak.nextcentury@gmail.com) wrote: > I noticed that in case of faulty startup the lone message about server connection is followed by messages from Tomcat: > > 2012-10-18 17:32:30,813 EDT [elasticsearch[Berzerker][generic][T#7]] DEBUG[org.elasticsearch.transport.netty]: [Berzerker] connected to node [[Dreadknight][fqLtglKzTJ6uit2aYA\_YAw][inet[/192.168.35.137:9300]]] > 2012-10-18 17:32:54,160 [Membership-MemberExpired.] INFO org.apache.catalina.tribes.group.interceptors.TcpFailureDetector- Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{127, 0, 0, 1}:4001,{127, 0, 0, 1},4001, alive=95012, securePort=-1, UDP Port=-1, id={-4 33 -44 -69 -27 -103 68 -21 -88 -63 -92 120 -64 -55 -100 -36 }, payload={}, command={}, domain={},]] message. Will verify. > 2012-10-18 17:32:54,162 [Membership-MemberExpired.] INFO org.apache.catalina.tribes.group.interceptors.TcpFailureDetector- Verification complete. Member still alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{127, 0, 0, 1}:4001,{127, 0, 0, 1},4001, alive=95012, securePort=-1, UDP Port=-1, id={-4 33 -44 -69 -27 -103 68 -21 -88 -63 -92 120 -64 -55 -100 -36 }, payload={}, command={}, domain={},]] > 2012-10-18 17:32:57,675 [Membership-MemberExpired.] INFO org.apache.catalina.tribes.group.interceptors.TcpFailureDetector- Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{127, 0, 0, 1}:4001,{127, 0, 0, 1},4001, alive=98395, securePort=-1, UDP Port=-1, id={-4 33 -44 -69 -27 -103 68 -21 -88 -63 -92 120 -64 -55 -100 -36 }, payload={}, command={}, domain={},]] message. Will verify. > 2012-10-18 17:32:57,680 [Membership-MemberExpired.] INFO org.apache.catalina.tribes.group.interceptors.TcpFailureDetector- Verification complete. Member still alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{127, 0, 0, 1}:4001,{127, 0, 0, 1},4001, alive=98395, securePort=-1, UDP Port=-1, id={-4 33 -44 -69 -27 -103 68 -21 -88 -63 -92 120 -64 -55 -100 -36 }, payload={}, command={}, domain={},]] > > I wonder whether there is a relationship between memberDisappeared message from Tomcat and inability of Elasticsearch nodes to form a cluster. > > Michael > > -- -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:07am UTC](https://discuss.elastic.co/t/node-does-not-join-the-cluster/9415/4 "2017-07-06T03:07:45Z")