Blocking the communication between 2 ElasticSearch severs in 4 nodes cluster leads to split brain

moti_umansky · August 17, 2014, 2:55pm

We have a 4 node cluster.

discovery.zen.minimum_master_nodes is 3.

When the master is intes1, and we block the communication between intes1
and intes3, we are getting a split brain.

C:\Procedures\New\ElasticSearch>curl
"intes1:9200/_cluster/health?pretty=true"
{
"cluster_name" : "play_clust1",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 4,
"number_of_data_nodes" : 4,
"active_primary_shards" : 5,
"active_shards" : 10,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

C:\Procedures\New\ElasticSearch>curl
"intes2:9200/_cluster/health?pretty=true"
{
"cluster_name" : "play_clust1",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 4,
"number_of_data_nodes" : 4,
"active_primary_shards" : 5,
"active_shards" : 10,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

C:\Procedures\New\ElasticSearch>curl
"intes3:9200/_cluster/health?pretty=true"
{
"cluster_name" : "play_clust1",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 4,
"number_of_data_nodes" : 4,
"active_primary_shards" : 5,
"active_shards" : 10,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

C:\Procedures\New\ElasticSearch>curl
"intes4:9200/_cluster/health?pretty=true"
{
"cluster_name" : "play_clust1",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 4,
"number_of_data_nodes" : 4,
"active_primary_shards" : 5,
"active_shards" : 10,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

C:\Procedures\New\ElasticSearch>
C:\Procedures\New\ElasticSearch>
C:\Procedures\New\ElasticSearch>C:\Procedures\New\ElasticSearch\curl.exe
"intes1:9200/_cat/master"
WQugEpLIQ3OL6qJGXibQWA INTES1 100.10.122.88 intes1

C:\Procedures\New\ElasticSearch>C:\Procedures\New\ElasticSearch\curl.exe
"intes2:9200/_cat/master"
WQugEpLIQ3OL6qJGXibQWA INTES1 100.10.122.88 intes1

C:\Procedures\New\ElasticSearch>C:\Procedures\New\ElasticSearch\curl.exe
"intes3:9200/_cat/master"
WQugEpLIQ3OL6qJGXibQWA INTES1 100.10.122.88 intes1

C:\Procedures\New\ElasticSearch>C:\Procedures\New\ElasticSearch\curl.exe
"intes4:9200/_cat/master"
WQugEpLIQ3OL6qJGXibQWA INTES1 100.10.122.88 intes1

C:\Procedures\New\ElasticSearch>

17:31 -> blocked communication between intes1 and intes3

C:\Procedures\New\ElasticSearch>curl
"intes1:9200/_cluster/health?pretty=true"
{
"cluster_name" : "play_clust1",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 4,
"number_of_data_nodes" : 4,
"active_primary_shards" : 5,
"active_shards" : 10,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

C:\Procedures\New\ElasticSearch>curl
"intes2:9200/_cluster/health?pretty=true"
{
"cluster_name" : "play_clust1",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 5,
"active_shards" : 10,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

C:\Procedures\New\ElasticSearch>curl
"intes3:9200/_cluster/health?pretty=true"
{
"cluster_name" : "play_clust1",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 5,
"active_shards" : 10,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

C:\Procedures\New\ElasticSearch>curl
"intes4:9200/_cluster/health?pretty=true"
{
"cluster_name" : "play_clust1",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 5,
"active_shards" : 10,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

C:\Procedures\New\ElasticSearch>C:\Procedures\New\ElasticSearch\curl.exe
"intes1:9200/_cat/master"
WQugEpLIQ3OL6qJGXibQWA INTES1 100.10.122.88 intes1

C:\Procedures\New\ElasticSearch>C:\Procedures\New\ElasticSearch\curl.exe
"intes2:9200/_cat/master"
TLU8vz2_SKmLCEIP0DQdeQ INTES3 100.10.122.90 intes3

C:\Procedures\New\ElasticSearch>C:\Procedures\New\ElasticSearch\curl.exe
"intes3:9200/_cat/master"
TLU8vz2_SKmLCEIP0DQdeQ INTES3 100.10.122.90 intes3

C:\Procedures\New\ElasticSearch>C:\Procedures\New\ElasticSearch\curl.exe
"intes4:9200/_cat/master"
TLU8vz2_SKmLCEIP0DQdeQ INTES3 100.10.122.90 intes3

I'll attach :

elasticsearch.yml
The ES logs from all the cluster nodes
The output of curl -XGET "intes1:9200/_nodes/?pretty=true"

Thanks,
Moti

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8673895f-888d-4c92-81fb-e7aa8a45f653%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

warkolm · August 17, 2014, 10:55pm

This looks similar to

github.com/elastic/elasticsearch

minimum_master_nodes does not prevent split-brain if splits are intersecting

opened 08:15AM - 17 Dec 12 UTC

closed 02:58PM - 01 Sep 14 UTC

saj

>bug v2.0.0-beta1 v1.4.0.Beta1

G'day, I'm using ElasticSearch 0.19.11 with the unicast Zen discovery protocol.… With this setup, I can easily split a 3-node cluster into two 'hemispheres' (continuing with the brain metaphor) with one node acting as a participant in both hemispheres. I believe this to be a significant problem, because now `minimum_master_nodes` is incapable of preventing certain split-brain scenarios. Here's what my 3-node test cluster looked like before I broke it: ![](https://saj.beta.anchortrove.com/es-splitbrain-1.png) Here's what the cluster looked like after simulating a communications failure between nodes (2) and (3): ![](https://saj.beta.anchortrove.com/es-splitbrain-2.png) Here's what seems to have happened immediately after the split: 1. Node (2) and (3) lose contact with one another. (`zen-disco-node_failed` ... `reason failed to ping`) 2. Node (2), still master of the left hemisphere, notes the disappearance of node (3) and broadcasts an advisory message to all of its followers. Node (1) takes note of the advisory. 3. Node (3) has now lost contact with its old master and decides to hold an election. It declares itself winner of the election. On declaring itself, it assumes master role of the right hemisphere, then broadcasts an advisory message to all of its followers. Node (1) takes note of this advisory, too. At this point, I can't say I know what to expect to find on node (1). If I query both masters for a list of nodes, I see node (1) in both clusters. Let's look at `minimum_master_nodes` as it applies to this test cluster. Assume I had set `minimum_master_nodes` to 2. Had node (3) been completely isolated from nodes (1) and (2), I would not have run into this problem. The left hemisphere would have enough nodes to satisfy the constraint; the right hemisphere would not. This would continue to work for larger clusters (with an appropriately larger value for `minimum_master_nodes`). The problem with `minimum_master_nodes` is that it does not work when the split brains are intersecting, as in my example above. Even on a larger cluster of, say, 7 nodes with `minimum_master_nodes` set to 4, all that needs to happen is for the 'right' two nodes to lose contact with one another (a master election has to take place) for the cluster to split. Is there anything that can be done to detect the intersecting split on node (1)? Would #1057 help? Am I missing something obvious? :)

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 18 August 2014 00:55, moti.umansky@gmail.com wrote:

We have a 4 node cluster.

discovery.zen.minimum_master_nodes is 3.

When the master is intes1, and we block the communication between intes1
and intes3, we are getting a split brain.

C:\Procedures\New\Elasticsearch>curl
"intes1:9200/_cluster/health?pretty=true"
{
"cluster_name" : "play_clust1",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 4,
"number_of_data_nodes" : 4,
"active_primary_shards" : 5,
"active_shards" : 10,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

C:\Procedures\New\Elasticsearch>curl
"intes2:9200/_cluster/health?pretty=true"
{
"cluster_name" : "play_clust1",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 4,
"number_of_data_nodes" : 4,
"active_primary_shards" : 5,
"active_shards" : 10,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

C:\Procedures\New\Elasticsearch>curl
"intes3:9200/_cluster/health?pretty=true"
{
"cluster_name" : "play_clust1",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 4,
"number_of_data_nodes" : 4,
"active_primary_shards" : 5,
"active_shards" : 10,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

C:\Procedures\New\Elasticsearch>curl
"intes4:9200/_cluster/health?pretty=true"
{
"cluster_name" : "play_clust1",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 4,
"number_of_data_nodes" : 4,
"active_primary_shards" : 5,
"active_shards" : 10,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

C:\Procedures\New\Elasticsearch>
C:\Procedures\New\Elasticsearch>
C:\Procedures\New\Elasticsearch>C:\Procedures\New\Elasticsearch\curl.exe
"intes1:9200/_cat/master"
WQugEpLIQ3OL6qJGXibQWA INTES1 100.10.122.88 intes1

C:\Procedures\New\Elasticsearch>C:\Procedures\New\Elasticsearch\curl.exe
"intes2:9200/_cat/master"
WQugEpLIQ3OL6qJGXibQWA INTES1 100.10.122.88 intes1

C:\Procedures\New\Elasticsearch>C:\Procedures\New\Elasticsearch\curl.exe
"intes3:9200/_cat/master"
WQugEpLIQ3OL6qJGXibQWA INTES1 100.10.122.88 intes1

C:\Procedures\New\Elasticsearch>C:\Procedures\New\Elasticsearch\curl.exe
"intes4:9200/_cat/master"
WQugEpLIQ3OL6qJGXibQWA INTES1 100.10.122.88 intes1

C:\Procedures\New\Elasticsearch>

17:31 -> blocked communication between intes1 and intes3

C:\Procedures\New\Elasticsearch>curl
"intes1:9200/_cluster/health?pretty=true"
{
"cluster_name" : "play_clust1",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 4,
"number_of_data_nodes" : 4,
"active_primary_shards" : 5,
"active_shards" : 10,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

C:\Procedures\New\Elasticsearch>curl
"intes2:9200/_cluster/health?pretty=true"
{
"cluster_name" : "play_clust1",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 5,
"active_shards" : 10,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

C:\Procedures\New\Elasticsearch>curl
"intes3:9200/_cluster/health?pretty=true"
{
"cluster_name" : "play_clust1",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 5,
"active_shards" : 10,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

C:\Procedures\New\Elasticsearch>curl
"intes4:9200/_cluster/health?pretty=true"
{
"cluster_name" : "play_clust1",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 5,
"active_shards" : 10,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

C:\Procedures\New\Elasticsearch>C:\Procedures\New\Elasticsearch\curl.exe
"intes1:9200/_cat/master"
WQugEpLIQ3OL6qJGXibQWA INTES1 100.10.122.88 intes1

C:\Procedures\New\Elasticsearch>C:\Procedures\New\Elasticsearch\curl.exe
"intes2:9200/_cat/master"
TLU8vz2_SKmLCEIP0DQdeQ INTES3 100.10.122.90 intes3

C:\Procedures\New\Elasticsearch>C:\Procedures\New\Elasticsearch\curl.exe
"intes3:9200/_cat/master"
TLU8vz2_SKmLCEIP0DQdeQ INTES3 100.10.122.90 intes3

C:\Procedures\New\Elasticsearch>C:\Procedures\New\Elasticsearch\curl.exe
"intes4:9200/_cat/master"
TLU8vz2_SKmLCEIP0DQdeQ INTES3 100.10.122.90 intes3

I'll attach :

elasticsearch.yml

The ES logs from all the cluster nodes

The output of curl -XGET "intes1:9200/_nodes/?pretty=true"

Thanks,
Moti

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8673895f-888d-4c92-81fb-e7aa8a45f653%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8673895f-888d-4c92-81fb-e7aa8a45f653%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624aEgXy3EJHb97CfDxaTReQd7SjzD4A8%3D5Czz-n9VXkOAA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.