# Split brain problem in 2 node elasticsearch cluster **URL:** \ **Category:** Elasticsearch\ **Created:** [May 4, 2015, 5:03am UTC](https://discuss.elastic.co/t/split-brain-problem-in-2-node-elasticsearch-cluster/23524 "2015-05-04T05:03:14Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Gourav\_H\_Dhelaria](https://avatars.discourse-cdn.com/v4/letter/g/7cd45c/32.png) [@Gourav\_H\_Dhelaria](https://discuss.elastic.co/u/Gourav_H_Dhelaria)\ **Post date:** [May 4, 2015, 5:03am UTC](https://discuss.elastic.co/t/split-brain-problem-in-2-node-elasticsearch-cluster/23524/1 "2015-05-04T05:03:14Z")
Version: 1.4. Say there are 2 nodes X and Y, both capable of becoming master. When network goes down, both nodes get disconnected from each other and assume the responsibility of master. When network is restored, they don't ping each other and form a cluster. Elasticsearch service has to be restarted on any one of the nodes for them to form a cluster. Even after they form a cluster, all primary shards remain on one node ( on which the service was restarted ), and all replica shards are on the other node. This document [http://www.elastic.co/guide/en/elasticsearch/guide/current/\_important\_configuration\_changes.html](http://www.elastic.co/guide/en/elasticsearch/guide/current/_important_configuration_changes.html) mentions that there has to be an uneven number of master eligible nodes. Queries: 1. Is there a way of avoiding split brain problem in 2 node cluster ? 2. After network is restored, shouldn't the nodes ping each other and form a cluster ? 3. After the service is restarted to form the cluster, why don't the primary shards get distributed on both the nodes ? Thanks, Gourav -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/d15234b3-0ea1-4390-b136-2f02f69cd3f5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d15234b3-0ea1-4390-b136-2f02f69cd3f5%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 4, 2015, 5:17am UTC](https://discuss.elastic.co/t/split-brain-problem-in-2-node-elasticsearch-cluster/23524/2 "2015-05-04T05:17:56Z")
1. Why are they becoming split anyway? GC, other load, network? 2. Not if they both think they are masters. 3. Are you running replicas? If so ES doesn't really differentiate between the two. On 4 May 2015 at 15:03, Gourav H Dhelaria [gouravdhelaria@gmail.com](mailto:gouravdhelaria@gmail.com) wrote: > Version: 1.4. > Say there are 2 nodes X and Y, both capable of becoming master. > When network goes down, both nodes get disconnected from each other and > assume the responsibility of master. > When network is restored, they don't ping each other and form a cluster. > > Elasticsearch service has to be restarted on any one of the nodes for them > to form a cluster. Even after they form a cluster, all primary shards > remain on one node ( on which the service was restarted ), and all replica > shards are on the other node. > > This document > > [http://www.elastic.co/guide/en/elasticsearch/guide/current/\_important\_configuration\_changes.html](http://www.elastic.co/guide/en/elasticsearch/guide/current/_important_configuration_changes.html) > > mentions that there has to be an uneven number of master eligible nodes. > > Queries: > > 1. Is there a way of avoiding split brain problem in 2 node cluster ? > > 2. After network is restored, shouldn't the nodes ping each other and form > a cluster ? > > 3. After the service is restarted to form the cluster, why don't the > primary shards get distributed on both the nodes ? > > Thanks, > > Gourav > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/d15234b3-0ea1-4390-b136-2f02f69cd3f5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d15234b3-0ea1-4390-b136-2f02f69cd3f5%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/d15234b3-0ea1-4390-b136-2f02f69cd3f5%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/d15234b3-0ea1-4390-b136-2f02f69cd3f5%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8TkHYhJ\_3-YYGBuRoXYF1XxDeRgs9c76tQwA8t-9nO%3DA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8TkHYhJ_3-YYGBuRoXYF1XxDeRgs9c76tQwA8t-9nO%3DA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Gourav\_H\_Dhelaria](https://avatars.discourse-cdn.com/v4/letter/g/7cd45c/32.png) [@Gourav\_H\_Dhelaria](https://discuss.elastic.co/u/Gourav_H_Dhelaria)\ **Post date:** [May 4, 2015, 5:27am UTC](https://discuss.elastic.co/t/split-brain-problem-in-2-node-elasticsearch-cluster/23524/3 "2015-05-04T05:27:13Z")
1. After network goes down, they loose communication with each other. After that, they are becoming split. 2. They both think they are masters. Even if they think they are masters, shouldn't the ping happen to see if there are other nodes in the cluster ? 3. Number of replicas is set to 1. If ES doesn't differentiate, why are some shards primary and others replica ? On Monday, 4 May 2015 10:48:24 UTC+5:30, Mark Walkom wrote: > 1. Why are they becoming split anyway? GC, other load, network? > 2. Not if they both think they are masters. > 3. Are you running replicas? If so ES doesn't really differentiate > between the two. > > On 4 May 2015 at 15:03, Gourav H Dhelaria \<[gouravd...@gmail.com](mailto:gouravd...@gmail.com) > \\> wrote: > > > Version: 1.4. > > Say there are 2 nodes X and Y, both capable of becoming master. > > When network goes down, both nodes get disconnected from each other and > > assume the responsibility of master. > > When network is restored, they don't ping each other and form a cluster. > > > > Elasticsearch service has to be restarted on any one of the nodes for > > them to form a cluster. Even after they form a cluster, all primary shards > > remain on one node ( on which the service was restarted ), and all replica > > shards are on the other node. > > > > This document > > > > [http://www.elastic.co/guide/en/elasticsearch/guide/current/\_important\_configuration\_changes.html](http://www.elastic.co/guide/en/elasticsearch/guide/current/_important_configuration_changes.html) > > > > mentions that there has to be an uneven number of master eligible nodes. > > > > Queries: > > > > 1. Is there a way of avoiding split brain problem in 2 node cluster ? > > > > 2. After network is restored, shouldn't the nodes ping each other and > > form a cluster ? > > > > 3. After the service is restarted to form the cluster, why don't the > > primary shards get distributed on both the nodes ? > > > > Thanks, > > > > Gourav > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/d15234b3-0ea1-4390-b136-2f02f69cd3f5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d15234b3-0ea1-4390-b136-2f02f69cd3f5%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/d15234b3-0ea1-4390-b136-2f02f69cd3f5%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/d15234b3-0ea1-4390-b136-2f02f69cd3f5%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/bf7a9953-bc87-4b96-843d-7bff5899855f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/bf7a9953-bc87-4b96-843d-7bff5899855f%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 4, 2015, 6:20am UTC](https://discuss.elastic.co/t/split-brain-problem-in-2-node-elasticsearch-cluster/23524/4 "2015-05-04T06:20:00Z")
Your nodes aren't in different DCs are they? If so this is why we don't support such setups, because ES is latency sensitive and these sorts of things can happen very easily when your network is unreliable. They don't try to ping other nodes because you only have two, and if they lose contact with one another then they both assume they are masters and create their own cluster. Masters don't ping other nodes at random and see if they should be joining a different cluster. Logically there is no difference between a primary and a replica shard, the only physical difference is a flag that tells the cluster state which is which. This is why ES will never assign a primary and it's applicable replica to the same node. You cannot get around the root of your problem unless you add another node to and set min masters to ensure a majority quorum. On 4 May 2015 at 15:27, Gourav H Dhelaria [gouravdhelaria@gmail.com](mailto:gouravdhelaria@gmail.com) wrote: > 1. After network goes down, they loose communication with each other. > After that, they are becoming split. > 2. They both think they are masters. Even if they think they are masters, > shouldn't the ping happen to see if there are other nodes in the cluster ? > 3. Number of replicas is set to 1. If ES doesn't differentiate, why are > some shards primary and others replica ? > > On Monday, 4 May 2015 10:48:24 UTC+5:30, Mark Walkom wrote: > > > 1. Why are they becoming split anyway? GC, other load, network? > > 2. Not if they both think they are masters. > > 3. Are you running replicas? If so ES doesn't really differentiate > > between the two. > > > > On 4 May 2015 at 15:03, Gourav H Dhelaria [gouravd...@gmail.com](mailto:gouravd...@gmail.com) wrote: > > > > > Version: 1.4. > > > Say there are 2 nodes X and Y, both capable of becoming master. > > > When network goes down, both nodes get disconnected from each other and > > > assume the responsibility of master. > > > When network is restored, they don't ping each other and form a cluster. > > > > > > Elasticsearch service has to be restarted on any one of the nodes for > > > them to form a cluster. Even after they form a cluster, all primary shards > > > remain on one node ( on which the service was restarted ), and all replica > > > shards are on the other node. > > > > > > This document > > > > > > [http://www.elastic.co/guide/en/elasticsearch/guide/current/\_important\_configuration\_changes.html](http://www.elastic.co/guide/en/elasticsearch/guide/current/_important_configuration_changes.html) > > > > > > mentions that there has to be an uneven number of master eligible nodes. > > > > > > Queries: > > > > > > 1. Is there a way of avoiding split brain problem in 2 node cluster ? > > > > > > 2. After network is restored, shouldn't the nodes ping each other and > > > form a cluster ? > > > > > > 3. After the service is restarted to form the cluster, why don't the > > > primary shards get distributed on both the nodes ? > > > > > > Thanks, > > > > > > Gourav > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/d15234b3-0ea1-4390-b136-2f02f69cd3f5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d15234b3-0ea1-4390-b136-2f02f69cd3f5%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/d15234b3-0ea1-4390-b136-2f02f69cd3f5%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/d15234b3-0ea1-4390-b136-2f02f69cd3f5%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/bf7a9953-bc87-4b96-843d-7bff5899855f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/bf7a9953-bc87-4b96-843d-7bff5899855f%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/bf7a9953-bc87-4b96-843d-7bff5899855f%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/bf7a9953-bc87-4b96-843d-7bff5899855f%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8nc1A\_vmQ\_vyG4fq2uNqFA9kZO%2BT\_Y4ed0e6wwPM7ztA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8nc1A_vmQ_vyG4fq2uNqFA9kZO%2BT_Y4ed0e6wwPM7ztA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Jason\_Wee](https://avatars.discourse-cdn.com/v4/letter/j/7ea924/32.png) [@Jason\_Wee](https://discuss.elastic.co/u/Jason_Wee)\ **Post date:** [May 4, 2015, 6:32am UTC](https://discuss.elastic.co/t/split-brain-problem-in-2-node-elasticsearch-cluster/23524/5 "2015-05-04T06:32:16Z")
why must you have only two nodes, would it be possible to add one more nodes so split brain will not become an issue? jason On Mon, May 4, 2015 at 2:20 PM, Mark Walkom [markwalkom@gmail.com](mailto:markwalkom@gmail.com) wrote: > Your nodes aren't in different DCs are they? If so this is why we don't > support such setups, because ES is latency sensitive and these sorts of > things can happen very easily when your network is unreliable. > > They don't try to ping other nodes because you only have two, and if they > lose contact with one another then they both assume they are masters and > create their own cluster. Masters don't ping other nodes at random and see > if they should be joining a different cluster. > > Logically there is no difference between a primary and a replica shard, > the only physical difference is a flag that tells the cluster state which > is which. This is why ES will never assign a primary and it's applicable > replica to the same node. > > You cannot get around the root of your problem unless you add another node > to and set min masters to ensure a majority quorum. > > On 4 May 2015 at 15:27, Gourav H Dhelaria [gouravdhelaria@gmail.com](mailto:gouravdhelaria@gmail.com) > wrote: > > > 1. After network goes down, they loose communication with each other. > > After that, they are becoming split. > > 2. They both think they are masters. Even if they think they are masters, > > shouldn't the ping happen to see if there are other nodes in the cluster ? > > 3. Number of replicas is set to 1. If ES doesn't differentiate, why are > > some shards primary and others replica ? > > > > On Monday, 4 May 2015 10:48:24 UTC+5:30, Mark Walkom wrote: > > > > > 1. Why are they becoming split anyway? GC, other load, network? > > > 2. Not if they both think they are masters. > > > 3. Are you running replicas? If so ES doesn't really differentiate > > > between the two. > > > > > > On 4 May 2015 at 15:03, Gourav H Dhelaria [gouravd...@gmail.com](mailto:gouravd...@gmail.com) wrote: > > > > > > > Version: 1.4. > > > > Say there are 2 nodes X and Y, both capable of becoming master. > > > > When network goes down, both nodes get disconnected from each other and > > > > assume the responsibility of master. > > > > When network is restored, they don't ping each other and form a cluster. > > > > > > > > Elasticsearch service has to be restarted on any one of the nodes for > > > > them to form a cluster. Even after they form a cluster, all primary shards > > > > remain on one node ( on which the service was restarted ), and all replica > > > > shards are on the other node. > > > > > > > > This document > > > > > > > > [http://www.elastic.co/guide/en/elasticsearch/guide/current/\_important\_configuration\_changes.html](http://www.elastic.co/guide/en/elasticsearch/guide/current/_important_configuration_changes.html) > > > > > > > > mentions that there has to be an uneven number of master eligible nodes. > > > > > > > > Queries: > > > > > > > > 1. Is there a way of avoiding split brain problem in 2 node cluster ? > > > > > > > > 2. After network is restored, shouldn't the nodes ping each other and > > > > form a cluster ? > > > > > > > > 3. After the service is restarted to form the cluster, why don't the > > > > primary shards get distributed on both the nodes ? > > > > > > > > Thanks, > > > > > > > > Gourav > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/d15234b3-0ea1-4390-b136-2f02f69cd3f5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d15234b3-0ea1-4390-b136-2f02f69cd3f5%40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/d15234b3-0ea1-4390-b136-2f02f69cd3f5%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/d15234b3-0ea1-4390-b136-2f02f69cd3f5%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/bf7a9953-bc87-4b96-843d-7bff5899855f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/bf7a9953-bc87-4b96-843d-7bff5899855f%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/bf7a9953-bc87-4b96-843d-7bff5899855f%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/bf7a9953-bc87-4b96-843d-7bff5899855f%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8nc1A\_vmQ\_vyG4fq2uNqFA9kZO%2BT\_Y4ed0e6wwPM7ztA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8nc1A_vmQ_vyG4fq2uNqFA9kZO%2BT_Y4ed0e6wwPM7ztA%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8nc1A\_vmQ\_vyG4fq2uNqFA9kZO%2BT\_Y4ed0e6wwPM7ztA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8nc1A_vmQ_vyG4fq2uNqFA9kZO%2BT_Y4ed0e6wwPM7ztA%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHO4itx2SuCKAZHA%2BYjLv4kYGJNN7srx6FVxWJ\_UzxzPWZ628w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHO4itx2SuCKAZHA%2BYjLv4kYGJNN7srx6FVxWJ_UzxzPWZ628w%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Gourav\_H\_Dhelaria](https://avatars.discourse-cdn.com/v4/letter/g/7cd45c/32.png) [@Gourav\_H\_Dhelaria](https://discuss.elastic.co/u/Gourav_H_Dhelaria)\ **Post date:** [May 5, 2015, 8:19am UTC](https://discuss.elastic.co/t/split-brain-problem-in-2-node-elasticsearch-cluster/23524/6 "2015-05-05T08:19:39Z")
Looks like the only way around this would be to add more nodes and set minimum masters to ensure a majority quorum. Thanks. Gourav On Monday, 4 May 2015 12:02:27 UTC+5:30, Jason Wee wrote: > why must you have only two nodes, would it be possible to add one more > nodes so split brain will not become an issue? > > jason > > On Mon, May 4, 2015 at 2:20 PM, Mark Walkom \<[markw...@gmail.com](mailto:markw...@gmail.com) > \\> wrote: > > > Your nodes aren't in different DCs are they? If so this is why we don't > > support such setups, because ES is latency sensitive and these sorts of > > things can happen very easily when your network is unreliable. > > > > They don't try to ping other nodes because you only have two, and if they > > lose contact with one another then they both assume they are masters and > > create their own cluster. Masters don't ping other nodes at random and see > > if they should be joining a different cluster. > > > > Logically there is no difference between a primary and a replica shard, > > the only physical difference is a flag that tells the cluster state which > > is which. This is why ES will never assign a primary and it's applicable > > replica to the same node. > > > > You cannot get around the root of your problem unless you add another > > node to and set min masters to ensure a majority quorum. > > > > On 4 May 2015 at 15:27, Gourav H Dhelaria \<[gouravd...@gmail.com](mailto:gouravd...@gmail.com) > > \\> wrote: > > > > > 1. After network goes down, they loose communication with each other. > > > After that, they are becoming split. > > > 2. They both think they are masters. Even if they think they are > > > masters, shouldn't the ping happen to see if there are other nodes in the > > > cluster ? > > > 3. Number of replicas is set to 1. If ES doesn't differentiate, why are > > > some shards primary and others replica ? > > > > > > On Monday, 4 May 2015 10:48:24 UTC+5:30, Mark Walkom wrote: > > > > > > > 1. Why are they becoming split anyway? GC, other load, network? > > > > 2. Not if they both think they are masters. > > > > 3. Are you running replicas? If so ES doesn't really differentiate > > > > between the two. > > > > > > > > On 4 May 2015 at 15:03, Gourav H Dhelaria [gouravd...@gmail.com](mailto:gouravd...@gmail.com) wrote: > > > > > > > > > Version: 1.4. > > > > > Say there are 2 nodes X and Y, both capable of becoming master. > > > > > When network goes down, both nodes get disconnected from each other > > > > > and assume the responsibility of master. > > > > > When network is restored, they don't ping each other and form a > > > > > cluster. > > > > > > > > > > Elasticsearch service has to be restarted on any one of the nodes for > > > > > them to form a cluster. Even after they form a cluster, all primary shards > > > > > remain on one node ( on which the service was restarted ), and all replica > > > > > shards are on the other node. > > > > > > > > > > This document > > > > > > > > > > [http://www.elastic.co/guide/en/elasticsearch/guide/current/\_important\_configuration\_changes.html](http://www.elastic.co/guide/en/elasticsearch/guide/current/_important_configuration_changes.html) > > > > > > > > > > mentions that there has to be an uneven number of master eligible > > > > > nodes. > > > > > > > > > > Queries: > > > > > > > > > > 1. Is there a way of avoiding split brain problem in 2 node cluster ? > > > > > > > > > > 2. After network is restored, shouldn't the nodes ping each other and > > > > > form a cluster ? > > > > > > > > > > 3. After the service is restarted to form the cluster, why don't the > > > > > primary shards get distributed on both the nodes ? > > > > > > > > > > Thanks, > > > > > > > > > > Gourav > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit > > > > > [https://groups.google.com/d/msgid/elasticsearch/d15234b3-0ea1-4390-b136-2f02f69cd3f5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d15234b3-0ea1-4390-b136-2f02f69cd3f5%40googlegroups.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/d15234b3-0ea1-4390-b136-2f02f69cd3f5%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/d15234b3-0ea1-4390-b136-2f02f69cd3f5%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/bf7a9953-bc87-4b96-843d-7bff5899855f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/bf7a9953-bc87-4b96-843d-7bff5899855f%40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/bf7a9953-bc87-4b96-843d-7bff5899855f%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/bf7a9953-bc87-4b96-843d-7bff5899855f%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8nc1A\_vmQ\_vyG4fq2uNqFA9kZO%2BT\_Y4ed0e6wwPM7ztA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8nc1A_vmQ_vyG4fq2uNqFA9kZO%2BT_Y4ed0e6wwPM7ztA%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8nc1A\_vmQ\_vyG4fq2uNqFA9kZO%2BT\_Y4ed0e6wwPM7ztA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8nc1A_vmQ_vyG4fq2uNqFA9kZO%2BT_Y4ed0e6wwPM7ztA%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/ba2982b3-fe59-4d09-8739-43c0384cf901%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ba2982b3-fe59-4d09-8739-43c0384cf901%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [May 5, 2015, 8:57am UTC](https://discuss.elastic.co/t/split-brain-problem-in-2-node-elasticsearch-cluster/23524/7 "2015-05-05T08:57:17Z")
In non "big data" scenarios, having two servers for a database is simply done to achieve high availability. Most databases use a master client scenario, but Elasticsearch does not support such a setup. It really should because not everyone has tons of data. Ivan, not affiliated with the OP On May 4, 2015 8:32 AM, "Jason Wee" [peichieh@gmail.com](mailto:peichieh@gmail.com) wrote: > why must you have only two nodes, would it be possible to add one more > nodes so split brain will not become an issue? > > jason > > On Mon, May 4, 2015 at 2:20 PM, Mark Walkom [markwalkom@gmail.com](mailto:markwalkom@gmail.com) wrote: > > > Your nodes aren't in different DCs are they? If so this is why we don't > > support such setups, because ES is latency sensitive and these sorts of > > things can happen very easily when your network is unreliable. > > > > They don't try to ping other nodes because you only have two, and if they > > lose contact with one another then they both assume they are masters and > > create their own cluster. Masters don't ping other nodes at random and see > > if they should be joining a different cluster. > > > > Logically there is no difference between a primary and a replica shard, > > the only physical difference is a flag that tells the cluster state which > > is which. This is why ES will never assign a primary and it's applicable > > replica to the same node. > > > > You cannot get around the root of your problem unless you add another > > node to and set min masters to ensure a majority quorum. > > > > On 4 May 2015 at 15:27, Gourav H Dhelaria [gouravdhelaria@gmail.com](mailto:gouravdhelaria@gmail.com) > > wrote: > > > > > 1. After network goes down, they loose communication with each other. > > > After that, they are becoming split. > > > 2. They both think they are masters. Even if they think they are > > > masters, shouldn't the ping happen to see if there are other nodes in the > > > cluster ? > > > 3. Number of replicas is set to 1. If ES doesn't differentiate, why are > > > some shards primary and others replica ? > > > > > > On Monday, 4 May 2015 10:48:24 UTC+5:30, Mark Walkom wrote: > > > > > > > 1. Why are they becoming split anyway? GC, other load, network? > > > > 2. Not if they both think they are masters. > > > > 3. Are you running replicas? If so ES doesn't really differentiate > > > > between the two. > > > > > > > > On 4 May 2015 at 15:03, Gourav H Dhelaria [gouravd...@gmail.com](mailto:gouravd...@gmail.com) wrote: > > > > > > > > > Version: 1.4. > > > > > Say there are 2 nodes X and Y, both capable of becoming master. > > > > > When network goes down, both nodes get disconnected from each other > > > > > and assume the responsibility of master. > > > > > When network is restored, they don't ping each other and form a > > > > > cluster. > > > > > > > > > > Elasticsearch service has to be restarted on any one of the nodes for > > > > > them to form a cluster. Even after they form a cluster, all primary shards > > > > > remain on one node ( on which the service was restarted ), and all replica > > > > > shards are on the other node. > > > > > > > > > > This document > > > > > > > > > > [http://www.elastic.co/guide/en/elasticsearch/guide/current/\_important\_configuration\_changes.html](http://www.elastic.co/guide/en/elasticsearch/guide/current/_important_configuration_changes.html) > > > > > > > > > > mentions that there has to be an uneven number of master eligible > > > > > nodes. > > > > > > > > > > Queries: > > > > > > > > > > 1. Is there a way of avoiding split brain problem in 2 node cluster ? > > > > > > > > > > 2. After network is restored, shouldn't the nodes ping each other and > > > > > form a cluster ? > > > > > > > > > > 3. After the service is restarted to form the cluster, why don't the > > > > > primary shards get distributed on both the nodes ? > > > > > > > > > > Thanks, > > > > > > > > > > Gourav > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit > > > > > [https://groups.google.com/d/msgid/elasticsearch/d15234b3-0ea1-4390-b136-2f02f69cd3f5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d15234b3-0ea1-4390-b136-2f02f69cd3f5%40googlegroups.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/d15234b3-0ea1-4390-b136-2f02f69cd3f5%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/d15234b3-0ea1-4390-b136-2f02f69cd3f5%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/bf7a9953-bc87-4b96-843d-7bff5899855f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/bf7a9953-bc87-4b96-843d-7bff5899855f%40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/bf7a9953-bc87-4b96-843d-7bff5899855f%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/bf7a9953-bc87-4b96-843d-7bff5899855f%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8nc1A\_vmQ\_vyG4fq2uNqFA9kZO%2BT\_Y4ed0e6wwPM7ztA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8nc1A_vmQ_vyG4fq2uNqFA9kZO%2BT_Y4ed0e6wwPM7ztA%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8nc1A\_vmQ\_vyG4fq2uNqFA9kZO%2BT\_Y4ed0e6wwPM7ztA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8nc1A_vmQ_vyG4fq2uNqFA9kZO%2BT_Y4ed0e6wwPM7ztA%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAHO4itx2SuCKAZHA%2BYjLv4kYGJNN7srx6FVxWJ\_UzxzPWZ628w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHO4itx2SuCKAZHA%2BYjLv4kYGJNN7srx6FVxWJ_UzxzPWZ628w%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAHO4itx2SuCKAZHA%2BYjLv4kYGJNN7srx6FVxWJ\_UzxzPWZ628w%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHO4itx2SuCKAZHA%2BYjLv4kYGJNN7srx6FVxWJ_UzxzPWZ628w%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQBXXO0h5Sa7LKaSirzvVnjoxHdu%3D3v4Pq0b\_MfUto%2BAbg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQBXXO0h5Sa7LKaSirzvVnjoxHdu%3D3v4Pq0b_MfUto%2BAbg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:15am UTC](https://discuss.elastic.co/t/split-brain-problem-in-2-node-elasticsearch-cluster/23524/8 "2017-07-06T00:15:43Z")