# Blocking the communication between 2 ElasticSearch severs in 4 nodes cluster leads to split brain **URL:** \ **Category:** Elasticsearch\ **Created:** [August 17, 2014, 2:55pm UTC](https://discuss.elastic.co/t/blocking-the-communication-between-2-elasticsearch-severs-in-4-nodes-cluster-leads-to-split-brain/19312 "2014-08-17T14:55:46Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![moti\_umansky](https://avatars.discourse-cdn.com/v4/letter/m/5f9b8f/32.png) [@moti\_umansky](https://discuss.elastic.co/u/moti_umansky)\ **Post date:** [August 17, 2014, 2:55pm UTC](https://discuss.elastic.co/t/blocking-the-communication-between-2-elasticsearch-severs-in-4-nodes-cluster-leads-to-split-brain/19312/1 "2014-08-17T14:55:46Z")
We have a 4 node cluster. discovery.zen.minimum\_master\_nodes is 3. When the master is intes1, and we block the communication between intes1 and intes3, we are getting a split brain. C:\Procedures\New\ElasticSearch\>curl "intes1:9200/\_cluster/health?pretty=true" { "cluster\_name" : "play\_clust1", "status" : "green", "timed\_out" : false, "number\_of\_nodes" : 4, "number\_of\_data\_nodes" : 4, "active\_primary\_shards" : 5, "active\_shards" : 10, "relocating\_shards" : 0, "initializing\_shards" : 0, "unassigned\_shards" : 0 } C:\Procedures\New\ElasticSearch\>curl "intes2:9200/\_cluster/health?pretty=true" { "cluster\_name" : "play\_clust1", "status" : "green", "timed\_out" : false, "number\_of\_nodes" : 4, "number\_of\_data\_nodes" : 4, "active\_primary\_shards" : 5, "active\_shards" : 10, "relocating\_shards" : 0, "initializing\_shards" : 0, "unassigned\_shards" : 0 } C:\Procedures\New\ElasticSearch\>curl "intes3:9200/\_cluster/health?pretty=true" { "cluster\_name" : "play\_clust1", "status" : "green", "timed\_out" : false, "number\_of\_nodes" : 4, "number\_of\_data\_nodes" : 4, "active\_primary\_shards" : 5, "active\_shards" : 10, "relocating\_shards" : 0, "initializing\_shards" : 0, "unassigned\_shards" : 0 } C:\Procedures\New\ElasticSearch\>curl "intes4:9200/\_cluster/health?pretty=true" { "cluster\_name" : "play\_clust1", "status" : "green", "timed\_out" : false, "number\_of\_nodes" : 4, "number\_of\_data\_nodes" : 4, "active\_primary\_shards" : 5, "active\_shards" : 10, "relocating\_shards" : 0, "initializing\_shards" : 0, "unassigned\_shards" : 0 } C:\Procedures\New\ElasticSearch\> C:\Procedures\New\ElasticSearch\> C:\Procedures\New\ElasticSearch\>C:\Procedures\New\ElasticSearch\curl.exe "intes1:9200/\_cat/master" WQugEpLIQ3OL6qJGXibQWA INTES1 100.10.122.88 intes1 C:\Procedures\New\ElasticSearch\>C:\Procedures\New\ElasticSearch\curl.exe "intes2:9200/\_cat/master" WQugEpLIQ3OL6qJGXibQWA INTES1 100.10.122.88 intes1 C:\Procedures\New\ElasticSearch\>C:\Procedures\New\ElasticSearch\curl.exe "intes3:9200/\_cat/master" WQugEpLIQ3OL6qJGXibQWA INTES1 100.10.122.88 intes1 C:\Procedures\New\ElasticSearch\>C:\Procedures\New\ElasticSearch\curl.exe "intes4:9200/\_cat/master" WQugEpLIQ3OL6qJGXibQWA INTES1 100.10.122.88 intes1 C:\Procedures\New\ElasticSearch\> 17:31 -\> blocked communication between intes1 and intes3 C:\Procedures\New\ElasticSearch\>curl "intes1:9200/\_cluster/health?pretty=true" { "cluster\_name" : "play\_clust1", "status" : "green", "timed\_out" : false, "number\_of\_nodes" : 4, "number\_of\_data\_nodes" : 4, "active\_primary\_shards" : 5, "active\_shards" : 10, "relocating\_shards" : 0, "initializing\_shards" : 0, "unassigned\_shards" : 0 } C:\Procedures\New\ElasticSearch\>curl "intes2:9200/\_cluster/health?pretty=true" { "cluster\_name" : "play\_clust1", "status" : "green", "timed\_out" : false, "number\_of\_nodes" : 3, "number\_of\_data\_nodes" : 3, "active\_primary\_shards" : 5, "active\_shards" : 10, "relocating\_shards" : 0, "initializing\_shards" : 0, "unassigned\_shards" : 0 } C:\Procedures\New\ElasticSearch\>curl "intes3:9200/\_cluster/health?pretty=true" { "cluster\_name" : "play\_clust1", "status" : "green", "timed\_out" : false, "number\_of\_nodes" : 3, "number\_of\_data\_nodes" : 3, "active\_primary\_shards" : 5, "active\_shards" : 10, "relocating\_shards" : 0, "initializing\_shards" : 0, "unassigned\_shards" : 0 } C:\Procedures\New\ElasticSearch\>curl "intes4:9200/\_cluster/health?pretty=true" { "cluster\_name" : "play\_clust1", "status" : "green", "timed\_out" : false, "number\_of\_nodes" : 3, "number\_of\_data\_nodes" : 3, "active\_primary\_shards" : 5, "active\_shards" : 10, "relocating\_shards" : 0, "initializing\_shards" : 0, "unassigned\_shards" : 0 } C:\Procedures\New\ElasticSearch\>C:\Procedures\New\ElasticSearch\curl.exe "intes1:9200/\_cat/master" WQugEpLIQ3OL6qJGXibQWA INTES1 100.10.122.88 intes1 C:\Procedures\New\ElasticSearch\>C:\Procedures\New\ElasticSearch\curl.exe "intes2:9200/\_cat/master" TLU8vz2\_SKmLCEIP0DQdeQ INTES3 100.10.122.90 intes3 C:\Procedures\New\ElasticSearch\>C:\Procedures\New\ElasticSearch\curl.exe "intes3:9200/\_cat/master" TLU8vz2\_SKmLCEIP0DQdeQ INTES3 100.10.122.90 intes3 C:\Procedures\New\ElasticSearch\>C:\Procedures\New\ElasticSearch\curl.exe "intes4:9200/\_cat/master" TLU8vz2\_SKmLCEIP0DQdeQ INTES3 100.10.122.90 intes3 I'll attach : - elasticsearch.yml - The ES logs from all the cluster nodes - The output of curl -XGET "intes1:9200/\_nodes/?pretty=true" Thanks, Moti -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/8673895f-888d-4c92-81fb-e7aa8a45f653%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8673895f-888d-4c92-81fb-e7aa8a45f653%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [August 17, 2014, 10:55pm UTC](https://discuss.elastic.co/t/blocking-the-communication-between-2-elasticsearch-severs-in-4-nodes-cluster-leads-to-split-brain/19312/2 "2014-08-17T22:55:28Z")
This looks similar to > > > G'day, > > I'm using ElasticSearch 0.19.11 with the unicast Zen discovery protocol.… > > With this setup, I can easily split a 3-node cluster into two 'hemispheres' (continuing with the brain metaphor) with one node acting as a participant in both hemispheres. I believe this to be a significant problem, because now \`minimum\_master\_nodes\` is incapable of preventing certain split-brain scenarios. > > Here's what my 3-node test cluster looked like before I broke it: > > !\[\](https://saj.beta.anchortrove.com/es-splitbrain-1.png) > > Here's what the cluster looked like after simulating a communications failure between nodes (2) and (3): > > !\[\](https://saj.beta.anchortrove.com/es-splitbrain-2.png) > > Here's what seems to have happened immediately after the split: > 1. Node (2) and (3) lose contact with one another. (\`zen-disco-node\_failed\` ... \`reason failed to ping\`) > 2. Node (2), still master of the left hemisphere, notes the disappearance of node (3) and broadcasts an advisory message to all of its followers. Node (1) takes note of the advisory. > 3. Node (3) has now lost contact with its old master and decides to hold an election. It declares itself winner of the election. On declaring itself, it assumes master role of the right hemisphere, then broadcasts an advisory message to all of its followers. Node (1) takes note of this advisory, too. > > At this point, I can't say I know what to expect to find on node (1). If I query both masters for a list of nodes, I see node (1) in both clusters. > > Let's look at \`minimum\_master\_nodes\` as it applies to this test cluster. Assume I had set \`minimum\_master\_nodes\` to 2. Had node (3) been completely isolated from nodes (1) and (2), I would not have run into this problem. The left hemisphere would have enough nodes to satisfy the constraint; the right hemisphere would not. This would continue to work for larger clusters (with an appropriately larger value for \`minimum\_master\_nodes\`). > > The problem with \`minimum\_master\_nodes\` is that it does not work when the split brains are intersecting, as in my example above. Even on a larger cluster of, say, 7 nodes with \`minimum\_master\_nodes\` set to 4, all that needs to happen is for the 'right' two nodes to lose contact with one another (a master election has to take place) for the cluster to split. > > Is there anything that can be done to detect the intersecting split on node (1)? > > Would #1057 help? > > Am I missing something obvious? :) Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 18 August 2014 00:55, [moti.umansky@gmail.com](mailto:moti.umansky@gmail.com) wrote: > We have a 4 node cluster. > > discovery.zen.minimum\_master\_nodes is 3. > > When the master is intes1, and we block the communication between intes1 > and intes3, we are getting a split brain. > > C:\Procedures\New\Elasticsearch\>curl > "intes1:9200/\_cluster/health?pretty=true" > { > "cluster\_name" : "play\_clust1", > "status" : "green", > "timed\_out" : false, > "number\_of\_nodes" : 4, > "number\_of\_data\_nodes" : 4, > "active\_primary\_shards" : 5, > "active\_shards" : 10, > "relocating\_shards" : 0, > "initializing\_shards" : 0, > "unassigned\_shards" : 0 > } > > C:\Procedures\New\Elasticsearch\>curl > "intes2:9200/\_cluster/health?pretty=true" > { > "cluster\_name" : "play\_clust1", > "status" : "green", > "timed\_out" : false, > "number\_of\_nodes" : 4, > "number\_of\_data\_nodes" : 4, > "active\_primary\_shards" : 5, > "active\_shards" : 10, > "relocating\_shards" : 0, > "initializing\_shards" : 0, > "unassigned\_shards" : 0 > } > > C:\Procedures\New\Elasticsearch\>curl > "intes3:9200/\_cluster/health?pretty=true" > { > "cluster\_name" : "play\_clust1", > "status" : "green", > "timed\_out" : false, > "number\_of\_nodes" : 4, > "number\_of\_data\_nodes" : 4, > "active\_primary\_shards" : 5, > "active\_shards" : 10, > "relocating\_shards" : 0, > "initializing\_shards" : 0, > "unassigned\_shards" : 0 > } > > C:\Procedures\New\Elasticsearch\>curl > "intes4:9200/\_cluster/health?pretty=true" > { > "cluster\_name" : "play\_clust1", > "status" : "green", > "timed\_out" : false, > "number\_of\_nodes" : 4, > "number\_of\_data\_nodes" : 4, > "active\_primary\_shards" : 5, > "active\_shards" : 10, > "relocating\_shards" : 0, > "initializing\_shards" : 0, > "unassigned\_shards" : 0 > } > > C:\Procedures\New\Elasticsearch\> > C:\Procedures\New\Elasticsearch\> > C:\Procedures\New\Elasticsearch\>C:\Procedures\New\Elasticsearch\curl.exe > "intes1:9200/\_cat/master" > WQugEpLIQ3OL6qJGXibQWA INTES1 100.10.122.88 intes1 > > C:\Procedures\New\Elasticsearch\>C:\Procedures\New\Elasticsearch\curl.exe > "intes2:9200/\_cat/master" > WQugEpLIQ3OL6qJGXibQWA INTES1 100.10.122.88 intes1 > > C:\Procedures\New\Elasticsearch\>C:\Procedures\New\Elasticsearch\curl.exe > "intes3:9200/\_cat/master" > WQugEpLIQ3OL6qJGXibQWA INTES1 100.10.122.88 intes1 > > C:\Procedures\New\Elasticsearch\>C:\Procedures\New\Elasticsearch\curl.exe > "intes4:9200/\_cat/master" > WQugEpLIQ3OL6qJGXibQWA INTES1 100.10.122.88 intes1 > > C:\Procedures\New\Elasticsearch\> > > 17:31 -\> blocked communication between intes1 and intes3 > > C:\Procedures\New\Elasticsearch\>curl > "intes1:9200/\_cluster/health?pretty=true" > { > "cluster\_name" : "play\_clust1", > "status" : "green", > "timed\_out" : false, > "number\_of\_nodes" : 4, > "number\_of\_data\_nodes" : 4, > "active\_primary\_shards" : 5, > "active\_shards" : 10, > "relocating\_shards" : 0, > "initializing\_shards" : 0, > "unassigned\_shards" : 0 > } > > C:\Procedures\New\Elasticsearch\>curl > "intes2:9200/\_cluster/health?pretty=true" > { > "cluster\_name" : "play\_clust1", > "status" : "green", > "timed\_out" : false, > "number\_of\_nodes" : 3, > "number\_of\_data\_nodes" : 3, > "active\_primary\_shards" : 5, > "active\_shards" : 10, > "relocating\_shards" : 0, > "initializing\_shards" : 0, > "unassigned\_shards" : 0 > } > > C:\Procedures\New\Elasticsearch\>curl > "intes3:9200/\_cluster/health?pretty=true" > { > "cluster\_name" : "play\_clust1", > "status" : "green", > "timed\_out" : false, > "number\_of\_nodes" : 3, > "number\_of\_data\_nodes" : 3, > "active\_primary\_shards" : 5, > "active\_shards" : 10, > "relocating\_shards" : 0, > "initializing\_shards" : 0, > "unassigned\_shards" : 0 > } > > C:\Procedures\New\Elasticsearch\>curl > "intes4:9200/\_cluster/health?pretty=true" > { > "cluster\_name" : "play\_clust1", > "status" : "green", > "timed\_out" : false, > "number\_of\_nodes" : 3, > "number\_of\_data\_nodes" : 3, > "active\_primary\_shards" : 5, > "active\_shards" : 10, > "relocating\_shards" : 0, > "initializing\_shards" : 0, > "unassigned\_shards" : 0 > } > > C:\Procedures\New\Elasticsearch\>C:\Procedures\New\Elasticsearch\curl.exe > "intes1:9200/\_cat/master" > WQugEpLIQ3OL6qJGXibQWA INTES1 100.10.122.88 intes1 > > C:\Procedures\New\Elasticsearch\>C:\Procedures\New\Elasticsearch\curl.exe > "intes2:9200/\_cat/master" > TLU8vz2\_SKmLCEIP0DQdeQ INTES3 100.10.122.90 intes3 > > C:\Procedures\New\Elasticsearch\>C:\Procedures\New\Elasticsearch\curl.exe > "intes3:9200/\_cat/master" > TLU8vz2\_SKmLCEIP0DQdeQ INTES3 100.10.122.90 intes3 > > C:\Procedures\New\Elasticsearch\>C:\Procedures\New\Elasticsearch\curl.exe > "intes4:9200/\_cat/master" > TLU8vz2\_SKmLCEIP0DQdeQ INTES3 100.10.122.90 intes3 > > I'll attach : > > - elasticsearch.yml > - The ES logs from all the cluster nodes > - The output of curl -XGET "intes1:9200/\_nodes/?pretty=true" > > Thanks, > Moti > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/8673895f-888d-4c92-81fb-e7aa8a45f653%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8673895f-888d-4c92-81fb-e7aa8a45f653%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/8673895f-888d-4c92-81fb-e7aa8a45f653%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8673895f-888d-4c92-81fb-e7aa8a45f653%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624aEgXy3EJHb97CfDxaTReQd7SjzD4A8%3D5Czz-n9VXkOAA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624aEgXy3EJHb97CfDxaTReQd7SjzD4A8%3D5Czz-n9VXkOAA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:08am UTC](https://discuss.elastic.co/t/blocking-the-communication-between-2-elasticsearch-severs-in-4-nodes-cluster-leads-to-split-brain/19312/3 "2017-07-06T01:08:06Z")