# Split-Brain Detection
**URL:** https://discuss.elastic.co/t/split-brain-detection/5140
**Category:** Elasticsearch
**Created:** [August 12, 2011, 3:19pm UTC](https://discuss.elastic.co/t/split-brain-detection/5140 "2011-08-12T15:19:41Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@James\_Cook](https://discuss.elastic.co/u/James_Cook)
#### Post date: [August 12, 2011, 3:19pm UTC](https://discuss.elastic.co/t/split-brain-detection/5140/1 "2011-08-12T15:19:41Z")
_Background_
We embed elasticsearch into our web application, and then let Amazon EC2
control the startup or teardown of our server instances based on load
metrics. The exact same web app is used across all instances, therefore each
new instance brings a new elasticsearch node into existence.
I am trying to bulletproof some scenarios, and just want to ask what the
best practice might be for some of these situations.
_Split-Brain_
Because there is no way to know how many servers will be running at any
particular time, I am unsure what I might be able to do to recognize
splitbrain scenarios and take servers down who are not part of the "primary"
cluster. My thoughts at the moment are a service that performs these steps:
1. Identify all nodes in the EC2 group using EC2 API.
2. Perform a cluster health check against all the nodes returned from
this list.
3. In a healthy cluster, the number of nodes identified in the ES list
will match the EC2 list.
4. In a split-brain scenario:
1. I identify each group of nodes using the ES stats.
2. I keep the largest group of nodes running.
3. Remove the other nodes from the load balancer using EC2 API.
4. I terminate the other nodes using EC2 API.
5. Amazon EC2 will bring up new servers to replace the ones I terminated
depending on load requirements.
If this works, it does so only because there is another API (EC2) I can
consult for a definitive list of all nodes in my cluster. I worry that this
process will be too slow because data loss and corruption can happen the
moment one of the split brain nodes is used.
Are there any other approaches I might take?
---
### Author:  [@Paul\_Smith](https://discuss.elastic.co/u/Paul_Smith)
#### Post date: [August 13, 2011, 3:34am UTC](https://discuss.elastic.co/t/split-brain-detection/5140/2 "2011-08-13T03:34:27Z")
Could you start your 'euthanising' script (couldn't think of a better word,
sorry) by progressively killing off nodes, waiting for the transition from
yellow-\> back to green as the shards re-replicate, then kill another one,
waiting long enough each time to ensure transition to yellow-\>green. After
a certain # node deaths, you'll be stuck at yellow because there's no not
enough nodes to satisfy all replica configurations, so after enough time
you've worked out you've gone just 1 too far, so need to restart 1 node?
I'm sure there's some mathematical way way though to compute the minimum #
nodes to at least satisfy a yellow status for you cluster (all primary
shards allocated, but no replicas available). this would become your
minimum\_master\_node setting perhaps? This is a bit gut feel, I haven't
done the maths on that one though, and is probably for large clusters with
lots of shards probably restrictive because it'll require too many masters
to satisfy that setting perhaps.
I'm very interested to see how you go with what ever solution you end up
going with. Let us know!
Paul Smith
---
### Author:  [@James\_Cook](https://discuss.elastic.co/u/James_Cook)
#### Post date: [August 13, 2011, 10:14am UTC](https://discuss.elastic.co/t/split-brain-detection/5140/3 "2011-08-13T10:14:48Z")
Hi Paul,
We currently do set the minimum number of instances in a cluster to the
number of replicas + 1. This allows us to initialize a cluster in green
state. Of course from that point forward Amazon is creating and destroying
nodes, so using minimum\_master\_node cannot be predicted ahead of time.
If we get a split brain however, we might end up with a cluster in yellow
state as a side-effect of dropping below this threshold. There isn't a way
around this, but no data is lost. I would image that the split brains will
immediately start turning some replicas into primary shards to compensate
for the primary shards they have lost.
At this time, I think it would be safe to drop all of the nodes involved in
the split-brains that I will be euthanizing. There wouldn't be any reason
that I can foresee to wait for a transition to green is there?
The biggest problem is making sure that the split-brain I will be
euthanizing will not receive any updates before I can take them down or
remove them from the load balancer. I wonder if minimum\_master\_node could
help me in some way if it could be controlled dynamically?
For example, as AWS brings a new node online or takes one down, I set the
minimum\_master\_node to floor(total\_nodes / 2) + 1? Maybe this would help me
make sure that an ES instance doesn't perform an update the moment
split-brain occurs. I am assuming that ES can react that fast to a drop
below min\_master\_node, but my hunch is it is tied to the zen heartbeat.
-- jim
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [August 13, 2011, 12:19pm UTC](https://discuss.elastic.co/t/split-brain-detection/5140/4 "2011-08-13T12:19:23Z")
Hi,
Yes, the minimum master nodes is the best way to guard against that. The
heartbeat to other nodes detect a node going down in an ordinal manner very
fast, and a node exiting abnormally almost as fast (broken socket). It can
be longer if the socket is left dangling.... .
Regarding changing the minimum master nodes dynamically, it can certainly
be enabled for people to set it through an API (a set cluster level settings
API is long overdue).
In theory, we can have something like automatic\_minimum\_master\_nodes,
that can be set to something like 3-quorum, and elasticsearch will
automatically set it to either a minimum of 3 or a quorum of all eligible
master nodes in the cluster, but, this effectively means that you have a
value of 3, because you can't tell if a node is leaving because you brought
it down intentionally, or because it is dead. So its not really possible
(just throwing it out there, in case you thought about it).
On Sat, Aug 13, 2011 at 1:14 PM, James Cook [jcook@tracermedia.com](mailto:jcook@tracermedia.com) wrote:
> Hi Paul,
>
> We currently do set the minimum number of instances in a cluster to the
> number of replicas + 1. This allows us to initialize a cluster in green
> state. Of course from that point forward Amazon is creating and destroying
> nodes, so using minimum\_master\_node cannot be predicted ahead of time.
>
> If we get a split brain however, we might end up with a cluster in yellow
> state as a side-effect of dropping below this threshold. There isn't a way
> around this, but no data is lost. I would image that the split brains will
> immediately start turning some replicas into primary shards to compensate
> for the primary shards they have lost.
>
> At this time, I think it would be safe to drop all of the nodes involved in
> the split-brains that I will be euthanizing. There wouldn't be any reason
> that I can foresee to wait for a transition to green is there?
>
> The biggest problem is making sure that the split-brain I will be
> euthanizing will not receive any updates before I can take them down or
> remove them from the load balancer. I wonder if minimum\_master\_node could
> help me in some way if it could be controlled dynamically?
>
> For example, as AWS brings a new node online or takes one down, I set the
> minimum\_master\_node to floor(total\_nodes / 2) + 1? Maybe this would help me
> make sure that an ES instance doesn't perform an update the moment
> split-brain occurs. I am assuming that ES can react that fast to a drop
> below min\_master\_node, but my hunch is it is tied to the zen heartbeat.
>
> -- jim
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [August 13, 2011, 12:21pm UTC](https://discuss.elastic.co/t/split-brain-detection/5140/5 "2011-08-13T12:21:09Z")
One more thing to note, you can potentially simplify this by having
dedicated (small) instances of just master nodes (node.master set to true,
node.data set to false). For example, 3. And let the data nodes (node.master
set to false, node.data set to true) come and go as they want.
On Sat, Aug 13, 2011 at 3:19 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote:
> Hi,
>
> Yes, the minimum master nodes is the best way to guard against that. The
> heartbeat to other nodes detect a node going down in an ordinal manner very
> fast, and a node exiting abnormally almost as fast (broken socket). It can
> be longer if the socket is left dangling.... .
>
> Regarding changing the minimum master nodes dynamically, it can
> certainly be enabled for people to set it through an API (a set cluster
> level settings API is long overdue).
>
> In theory, we can have something like automatic\_minimum\_master\_nodes,
> that can be set to something like 3-quorum, and elasticsearch will
> automatically set it to either a minimum of 3 or a quorum of all eligible
> master nodes in the cluster, but, this effectively means that you have a
> value of 3, because you can't tell if a node is leaving because you brought
> it down intentionally, or because it is dead. So its not really possible
> (just throwing it out there, in case you thought about it).
>
> On Sat, Aug 13, 2011 at 1:14 PM, James Cook [jcook@tracermedia.com](mailto:jcook@tracermedia.com) wrote:
>
> > Hi Paul,
> >
> > We currently do set the minimum number of instances in a cluster to the
> > number of replicas + 1. This allows us to initialize a cluster in green
> > state. Of course from that point forward Amazon is creating and destroying
> > nodes, so using minimum\_master\_node cannot be predicted ahead of time.
> >
> > If we get a split brain however, we might end up with a cluster in yellow
> > state as a side-effect of dropping below this threshold. There isn't a way
> > around this, but no data is lost. I would image that the split brains will
> > immediately start turning some replicas into primary shards to compensate
> > for the primary shards they have lost.
> >
> > At this time, I think it would be safe to drop all of the nodes involved
> > in the split-brains that I will be euthanizing. There wouldn't be any reason
> > that I can foresee to wait for a transition to green is there?
> >
> > The biggest problem is making sure that the split-brain I will be
> > euthanizing will not receive any updates before I can take them down or
> > remove them from the load balancer. I wonder if minimum\_master\_node could
> > help me in some way if it could be controlled dynamically?
> >
> > For example, as AWS brings a new node online or takes one down, I set the
> > minimum\_master\_node to floor(total\_nodes / 2) + 1? Maybe this would help me
> > make sure that an ES instance doesn't perform an update the moment
> > split-brain occurs. I am assuming that ES can react that fast to a drop
> > below min\_master\_node, but my hunch is it is tied to the zen heartbeat.
> >
> > -- jim
---
### Author:  [@James\_Cook](https://discuss.elastic.co/u/James_Cook)
#### Post date: [August 13, 2011, 7:55pm UTC](https://discuss.elastic.co/t/split-brain-detection/5140/6 "2011-08-13T19:55:16Z")
One more thing to note, you can potentially simplify this by having
dedicated (small) instances of just master nodes (node.master set to true,
node.data set to false). For example, 3. And let the data nodes (node.master
set to false, node.data set to true) come and go as they want.
That's something I hadn't considered. Glad there are some alternative
approaches as well.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 3:57am UTC](https://discuss.elastic.co/t/split-brain-detection/5140/7 "2017-07-06T03:57:06Z")