# Split-brain situation - forcing discovery and rejoin **URL:** \ **Category:** Elasticsearch\ **Created:** [August 28, 2013, 12:00am UTC](https://discuss.elastic.co/t/split-brain-situation-forcing-discovery-and-rejoin/13355 "2013-08-28T00:00:10Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Andras\_Palinkas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andras_palinkas/32/2145_2.png) [@Andras\_Palinkas](https://discuss.elastic.co/u/Andras_Palinkas)\ **Post date:** [August 28, 2013, 12:00am UTC](https://discuss.elastic.co/t/split-brain-situation-forcing-discovery-and-rejoin/13355/1 "2013-08-28T00:00:10Z")
Hello ES-Team, I have the following situation. I'm using 6 ES nodes with unicast (unfortunately I cannot use multicast), I have discovery pings and timeouts set up (as defaults for now, 3 pings, 30 sec timeouts). Yesterday we had a network issue for more than 3 minutes and that caused that the ES cluster fell into 4 pieces (my config required only one master-able node which is also unfortunate, whatever...). In the logs I've seen that ES nodes decided to split the cluster after the 3rd ping. The problem is that once the network came back up, the ES nodes did not try to rejoin. How should this work? Is there any option that I could set so if the cluster is not complete (not each node is connected) retry in every X seconds to rejoin all whole cluster (use case: unicast, each node knows about all of the cluster nodes)? If there isn't any option like that, is there any way (for example calling an admin REST url) to force the rejoin of the cluster? I know that this could be done with restarting each nodes (running the ES with a through a process manager and calling the \_shutdown REST url for each node), but I don't really want to restart the nodes if it's not a must. Thanks in advance, Andras -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![btiernay](https://avatars.discourse-cdn.com/v4/letter/b/bb73d2/32.png) [@btiernay](https://discuss.elastic.co/u/btiernay)\ **Post date:** [August 28, 2013, 1:26am UTC](https://discuss.elastic.co/t/split-brain-situation-forcing-discovery-and-rejoin/13355/2 "2013-08-28T01:26:12Z")
See > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. "The discovery.zen.minimum\_master\_nodes allows to control the minimum number of master eligible nodes a node should “see” in order to operate within the cluster. Its recommended to set it to a higher value than 1 when running more than 2 nodes in the cluster." the best practice is to set it to at least N/2 + 1, but keep in mind: [http://elasticsearch-users.115913.n3.nabble.com/minimum-master-node-not-working-as-expected-td4037027.html](http://elasticsearch-users.115913.n3.nabble.com/minimum-master-node-not-working-as-expected-td4037027.html) On Tuesday, 27 August 2013 20:00:10 UTC-4, András Pálinkás wrote: > Hello ES-Team, > > I have the following situation. I'm using 6 ES nodes with unicast > (unfortunately I cannot use multicast), I have discovery pings and timeouts > set up (as defaults for now, 3 pings, 30 sec timeouts). > Yesterday we had a network issue for more than 3 minutes and that caused > that the ES cluster fell into 4 pieces (my config required only one > master-able node which is also unfortunate, whatever...). > In the logs I've seen that ES nodes decided to split the cluster after the > 3rd ping. > > The problem is that once the network came back up, the ES nodes did not > try to rejoin. > How should this work? Is there any option that I could set so if the > cluster is not complete (not each node is connected) retry in every X > seconds to rejoin all whole cluster (use case: unicast, each node knows > about all of the cluster nodes)? > > If there isn't any option like that, is there any way (for example calling > an admin REST url) to force the rejoin of the cluster? > I know that this could be done with restarting each nodes (running the ES > with a through a process manager and calling the \_shutdown REST url for > each node), but I don't really want to restart the nodes if it's not a must. > > Thanks in advance, > Andras -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Runar\_Myklebust](https://avatars.discourse-cdn.com/v4/letter/r/a6a055/32.png) [@Runar\_Myklebust](https://discuss.elastic.co/u/Runar_Myklebust)\ **Post date:** [August 28, 2013, 6:54am UTC](https://discuss.elastic.co/t/split-brain-situation-forcing-discovery-and-rejoin/13355/3 "2013-08-28T06:54:49Z")
This behavior is really a problem in certain scenarios. For a lot of our customers, its not an option to need to have N/2+1 nodes running for the system to be available, they want the system to be up running with e.g 1 of 14 nodes. I would really like to see a future option to force nodes to try rejoining cluster if any other nodes found in e.g unicast list. On Wed, Aug 28, 2013 at 3:26 AM, btiernay [rtiernay@gmail.com](mailto:rtiernay@gmail.com) wrote: > See > > [Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/guide/reference/modules/discovery/zen/) > > "The discovery.zen.minimum\_master\_nodes allows to control the minimum > number of master eligible nodes a node should “see” in order to operate > within the cluster. Its recommended to set it to a higher value than 1 when > running more than 2 nodes in the cluster." > > the best practice is to set it to at least N/2 + 1, but keep in mind: > > [http://elasticsearch-users.115913.n3.nabble.com/minimum-master-node-not-working-as-expected-td4037027.html](http://elasticsearch-users.115913.n3.nabble.com/minimum-master-node-not-working-as-expected-td4037027.html) > > On Tuesday, 27 August 2013 20:00:10 UTC-4, András Pálinkás wrote: > > > Hello ES-Team, > > > > I have the following situation. I'm using 6 ES nodes with unicast > > (unfortunately I cannot use multicast), I have discovery pings and timeouts > > set up (as defaults for now, 3 pings, 30 sec timeouts). > > Yesterday we had a network issue for more than 3 minutes and that caused > > that the ES cluster fell into 4 pieces (my config required only one > > master-able node which is also unfortunate, whatever...). > > In the logs I've seen that ES nodes decided to split the cluster after > > the 3rd ping. > > > > The problem is that once the network came back up, the ES nodes did not > > try to rejoin. > > How should this work? Is there any option that I could set so if the > > cluster is not complete (not each node is connected) retry in every X > > seconds to rejoin all whole cluster (use case: unicast, each node knows > > about all of the cluster nodes)? > > > > If there isn't any option like that, is there any way (for example > > calling an admin REST url) to force the rejoin of the cluster? > > I know that this could be done with restarting each nodes (running the ES > > with a through a process manager and calling the \_shutdown REST url for > > each node), but I don't really want to restart the nodes if it's not a must. > > > > Thanks in advance, > > Andras > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ## -- mvh Runar Myklebust -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:19am UTC](https://discuss.elastic.co/t/split-brain-situation-forcing-discovery-and-rejoin/13355/4 "2017-07-06T02:19:22Z")