# Node failing to join the cluster after reboot **URL:** \ **Category:** Elasticsearch\ **Created:** [October 16, 2013, 3:00pm UTC](https://discuss.elastic.co/t/node-failing-to-join-the-cluster-after-reboot/13972 "2013-10-16T15:00:58Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Joe\_Adams](https://avatars.discourse-cdn.com/v4/letter/j/aeb1de/32.png) [@Joe\_Adams](https://discuss.elastic.co/u/Joe_Adams)\ **Post date:** [October 16, 2013, 3:00pm UTC](https://discuss.elastic.co/t/node-failing-to-join-the-cluster-after-reboot/13972/1 "2013-10-16T15:00:58Z")
So I've been working with an elasticsearch cluster for a couple months now. I'm finally getting it into production, but after a soft launch I realized that I needed to allocate more ram to each instance. I'm running 3 boxes with 3 instances of elasticsearch each. I took the first box down, added the ram, and brought it back up into the cluster. All is well. Moving to the second, I now have a routing/load balancing node that won't come back into the cluster. The other 2 instances joined fine. I tried several times to reboot the failed instance with no luck. I keep getting a "no masterNode returned" error. Releveant Info 3 Machines, 3 Instances each Instance 1 + 2: Data + eligible master Instance 3: No data, no master (load balancing/routing only) OS: Centos 6.4 Elasticsearch version 0.90.3 ( I know this is somewhat dated now, we must extensively test new releases in dev/test before moving to production) IPTables: # Elasticsearch Rest API (HTTP) -A INPUT -m state --state NEW -m tcp -p tcp --dport 9200 -j ACCEPT #Elasticserach Transport Service -A INPUT -m state --state NEW -m tcp -p tcp --dport 9300 -j ACCEPT -A INPUT -m state --state NEW -m tcp -p tcp --dport 9301 -j ACCEPT -A INPUT -m state --state NEW -m tcp -p tcp --dport 9302 -j ACCEPT # Allow Mulitcast for ElasticSearch auto-discovery -A INPUT -m pkttype --pkt-type multicast -j ACCEPT Trace discovery logs: [https://gist.github.com/jumpinjoeadams/7008972](https://gist.github.com/jumpinjoeadams/7008972) Relevant ES config: cluster.name: NightRunnerProd http.enabled: false ( on instances 1+2 only) gateway.recover\_after\_nodes: 4 gateway.recover\_after\_time: 20s gateway.expected\_nodes: 6 ( These recovery options were lowered to resolve this issue previously, but it just prolonged the issue apparently) discovery.zen.minimum\_master\_nodes: 2 discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: ["10.11.253.173[9300-9305]", "10.11.253.174[9300-9305]", "10.11.253.175[9300-9305]"] node.master: false ( instance 3 only) node.data: false (instance 3 only) I have tried disabling iptables SELinux has no errors Google provides no help. Every time the node comes up, it doesn't join the cluster, just gives the 503. The failed node is the balancing node on machine 2. I'm going nuts trying to figure out why this happens from time to time. Thanks in advance! Joe -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Joe\_Adams](https://avatars.discourse-cdn.com/v4/letter/j/aeb1de/32.png) [@Joe\_Adams](https://discuss.elastic.co/u/Joe_Adams)\ **Post date:** [October 23, 2013, 2:51pm UTC](https://discuss.elastic.co/t/node-failing-to-join-the-cluster-after-reboot/13972/2 "2013-10-23T14:51:49Z")
It has been a week and I have still had no luck. Anyone have any ideas? I'm at a loss for ideas. On Wednesday, October 16, 2013 11:00:58 AM UTC-4, Joe Adams wrote: > So I've been working with an elasticsearch cluster for a couple months > now. I'm finally getting it into production, but after a soft launch I > realized that I needed to allocate more ram to each instance. I'm running 3 > boxes with 3 instances of elasticsearch each. I took the first box down, > added the ram, and brought it back up into the cluster. All is well. Moving > to the second, I now have a routing/load balancing node that won't come > back into the cluster. The other 2 instances joined fine. I tried several > times to reboot the failed instance with no luck. I keep getting a "no > masterNode returned" error. > > Releveant Info > > 3 Machines, 3 Instances each > Instance 1 + 2: Data + eligible master > Instance 3: No data, no master (load balancing/routing only) > > OS: Centos 6.4 > Elasticsearch version 0.90.3 ( I know this is somewhat dated now, we must > extensively test new releases in dev/test before moving to production) > > IPTables: > > # Elasticsearch Rest API (HTTP) > > -A INPUT -m state --state NEW -m tcp -p tcp --dport 9200 -j ACCEPT > #Elasticserach Transport Service > -A INPUT -m state --state NEW -m tcp -p tcp --dport 9300 -j ACCEPT > -A INPUT -m state --state NEW -m tcp -p tcp --dport 9301 -j ACCEPT > -A INPUT -m state --state NEW -m tcp -p tcp --dport 9302 -j ACCEPT > > # Allow Mulitcast for Elasticsearch auto-discovery > > -A INPUT -m pkttype --pkt-type multicast -j ACCEPT > > Trace discovery logs: > [https://gist.github.com/jumpinjoeadams/7008972](https://gist.github.com/jumpinjoeadams/7008972) > > Relevant ES config: > cluster.name: NightRunnerProd > http.enabled: false ( on instances 1+2 only) > gateway.recover\_after\_nodes: 4 > gateway.recover\_after\_time: 20s > gateway.expected\_nodes: 6 ( These recovery options were lowered to > resolve this issue previously, but it just prolonged the issue apparently) > discovery.zen.minimum\_master\_nodes: 2 > discovery.zen.ping.multicast.enabled: false > discovery.zen.ping.unicast.hosts: ["10.11.253.173[9300-9305]", > "10.11.253.174[9300-9305]", "10.11.253.175[9300-9305]"] > node.master: false ( instance 3 only) > node.data: false (instance 3 only) > > I have tried disabling iptables > SELinux has no errors > Google provides no help. > Every time the node comes up, it doesn't join the cluster, just gives the > 503. The failed node is the balancing node on machine 2. > I'm going nuts trying to figure out why this happens from time to time. > > Thanks in advance! > Joe -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [October 23, 2013, 3:20pm UTC](https://discuss.elastic.co/t/node-failing-to-join-the-cluster-after-reboot/13972/3 "2013-10-23T15:20:00Z")
Not sure what is going on. Which node is now master? gateway.expected\_nodes: 6 means, the cluster is waiting for 6 nodes. But afaik you have 3 nodes. Is this true? A node which is non-master eligible can complicate things (for a simple mind like me). discovery.zen.minimum\_master\_nodes: 2 can be quite optimistic. If you have three nodes and one node is non-master eligible, I would have chosen discovery.zen.minimum\_master\_nodes: 3 so all nodes must be up before master is elected. Just my 2p Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Joe\_Adams](https://avatars.discourse-cdn.com/v4/letter/j/aeb1de/32.png) [@Joe\_Adams](https://discuss.elastic.co/u/Joe_Adams)\ **Post date:** [October 23, 2013, 3:24pm UTC](https://discuss.elastic.co/t/node-failing-to-join-the-cluster-after-reboot/13972/4 "2013-10-23T15:24:43Z")
The master is NightRunner103.Instance2. There are 9 total nodes. 6 are data and 3 are non data. I will change my minimum master nodes and see if that helps. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [October 23, 2013, 3:29pm UTC](https://discuss.elastic.co/t/node-failing-to-join-the-cluster-after-reboot/13972/5 "2013-10-23T15:29:19Z")
If you have 9 nodes, you should take care to set minimum master nodes at least to 5 (half the cluster nodes plus 1) Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Joe\_Adams](https://avatars.discourse-cdn.com/v4/letter/j/aeb1de/32.png) [@Joe\_Adams](https://discuss.elastic.co/u/Joe_Adams)\ **Post date:** [October 23, 2013, 3:42pm UTC](https://discuss.elastic.co/t/node-failing-to-join-the-cluster-after-reboot/13972/6 "2013-10-23T15:42:54Z")
The 3 non data are not eligible masters though. They are set as load balancing. Is there a downside to changing these nodes to be eligible masters? That would give me an odd number of nodes and help prevent split brain. On Wednesday, October 23, 2013 11:29:19 AM UTC-4, Jörg Prante wrote: > If you have 9 nodes, you should take care to set minimum master nodes at > least to 5 (half the cluster nodes plus 1) > > Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ashish\_Nigam\_3](https://avatars.discourse-cdn.com/v4/letter/a/f19dbf/32.png) [@Ashish\_Nigam\_3](https://discuss.elastic.co/u/Ashish_Nigam_3)\ **Post date:** [October 23, 2013, 4:04pm UTC](https://discuss.elastic.co/t/node-failing-to-join-the-cluster-after-reboot/13972/7 "2013-10-23T16:04:13Z")
Is there are any downside to having nodes that have only one role - master, data or client? For example, say, I have a cluster where I have 3 masters, 5 data nodes and 2 client nodes. 5 data nodes and 2 client nodes are not master eligible and 2 client nodes are only meant for load balancing client requests. And in this case, I set minimum master nodes to be 2. Do you see any problem with this configuration? On Wed, Oct 23, 2013 at 8:29 AM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > If you have 9 nodes, you should take care to set minimum master nodes at > least to 5 (half the cluster nodes plus 1) > > Jörg > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Joe\_Adams](https://avatars.discourse-cdn.com/v4/letter/j/aeb1de/32.png) [@Joe\_Adams](https://discuss.elastic.co/u/Joe_Adams)\ **Post date:** [October 23, 2013, 6:12pm UTC](https://discuss.elastic.co/t/node-failing-to-join-the-cluster-after-reboot/13972/8 "2013-10-23T18:12:59Z")
I just modified my settings so that it expects 9 nodes, recovers after 6, minimum master is 5, and all nodes are eligible masters. After some testing, I have the same problem. I shutdown 3 at a time, everything came back up fine. I tested a large failure where I shutdown 6 / 9 nodes and when I brought them all back up, now 2 won't join the cluster. The cluster is green because all shards and replicas are ok again, but I have 2 missing nodes that won't join. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Joe\_Adams](https://avatars.discourse-cdn.com/v4/letter/j/aeb1de/32.png) [@Joe\_Adams](https://discuss.elastic.co/u/Joe_Adams)\ **Post date:** [October 23, 2013, 7:05pm UTC](https://discuss.elastic.co/t/node-failing-to-join-the-cluster-after-reboot/13972/9 "2013-10-23T19:05:09Z")
It looks like after some time and restarting the failed nodes again they were finally able to rejoin the cluster. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:10am UTC](https://discuss.elastic.co/t/node-failing-to-join-the-cluster-after-reboot/13972/10 "2017-07-06T02:10:59Z")