# Strange split brain scenario

**URL:** <https://discuss.elastic.co/t/strange-split-brain-scenario/5782>\
**Category:** Elasticsearch\
**Created:** [November 6, 2011, 4:18pm UTC](https://discuss.elastic.co/t/strange-split-brain-scenario/5782 "2011-11-06T16:18:49Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Eran\_Kutner\_2](https://avatars.discourse-cdn.com/v4/letter/e/dc4da7/32.png) [@Eran\_Kutner\_2](https://discuss.elastic.co/u/Eran_Kutner_2)\
**Post date:** [November 6, 2011, 4:18pm UTC](https://discuss.elastic.co/t/strange-split-brain-scenario/5782/1 "2011-11-06T16:18:49Z")

</div>

Hi,  
We're using version 0.17.6 with two servers and had a strange problem. One  
of the server identified itself (es1-01) as the master and the peer  
(es1-02) as the slave:  
{  
"cluster\_name" : "gcs",  
"master\_node" : "j0VcAKNRSsKnbHyaeof6pQ",  
"blocks" : {  
},  
"nodes" : {  
"zi6VNqehTCaZjNS0nbGUhg" : {  
"name" : "es1-02",  
"transport\_address" : "inet[/10.1.101.152:9300]",  
"attributes" : {  
}  
},  
"j0VcAKNRSsKnbHyaeof6pQ" : {  
"name" : "es1-01",  
"transport\_address" : "inet[/10.1.101.151:9300]",  
"attributes" : {  
}  
}  
},

While the other one only saw itself:  
{  
"cluster\_name" : "gcs",  
"master\_node" : "zi6VNqehTCaZjNS0nbGUhg",  
"blocks" : {  
},  
"nodes" : {  
"zi6VNqehTCaZjNS0nbGUhg" : {  
"name" : "es1-02",  
"transport\_address" : "inet[/10.1.101.152:9300]",  
"attributes" : {  
}  
}  
},

Only resetting es1-02 caused it to properly identify the other server.

Now es1-01 is the master of all the shards in all the indexes, and 3 days  
after the problem it didn't rebalance, is that expected? Is there a way to  
force rebalancing?

Thanks,  
Eran

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [November 9, 2011, 6:17am UTC](https://discuss.elastic.co/t/strange-split-brain-scenario/5782/2 "2011-11-09T06:17:09Z")

</div>

Do you have the logs from those two servers?

On Sun, Nov 6, 2011 at 6:18 PM, Eran Kutner [eran@gigya-inc.com](mailto:eran@gigya-inc.com) wrote:

> Hi,  
> We're using version 0.17.6 with two servers and had a strange problem. One  
> of the server identified itself (es1-01) as the master and the peer  
> (es1-02) as the slave:  
> {  
> "cluster\_name" : "gcs",  
> "master\_node" : "j0VcAKNRSsKnbHyaeof6pQ",  
> "blocks" : {  
> },  
> "nodes" : {  
> "zi6VNqehTCaZjNS0nbGUhg" : {  
> "name" : "es1-02",  
> "transport\_address" : "inet[/10.1.101.152:9300]",  
> "attributes" : {  
> }  
> },  
> "j0VcAKNRSsKnbHyaeof6pQ" : {  
> "name" : "es1-01",  
> "transport\_address" : "inet[/10.1.101.151:9300]",  
> "attributes" : {  
> }  
> }  
> },
> 
> While the other one only saw itself:  
> {  
> "cluster\_name" : "gcs",  
> "master\_node" : "zi6VNqehTCaZjNS0nbGUhg",  
> "blocks" : {  
> },  
> "nodes" : {  
> "zi6VNqehTCaZjNS0nbGUhg" : {  
> "name" : "es1-02",  
> "transport\_address" : "inet[/10.1.101.152:9300]",  
> "attributes" : {  
> }  
> }  
> },
> 
> Only resetting es1-02 caused it to properly identify the other server.
> 
> Now es1-01 is the master of all the shards in all the indexes, and 3 days  
> after the problem it didn't rebalance, is that expected? Is there a way to  
> force rebalancing?
> 
> Thanks,  
> Eran

---

<div class="post-metadata">

**Author:** ![Eran\_Kutner\_2](https://avatars.discourse-cdn.com/v4/letter/e/dc4da7/32.png) [@Eran\_Kutner\_2](https://discuss.elastic.co/u/Eran_Kutner_2)\
**Post date:** [November 14, 2011, 10:34am UTC](https://discuss.elastic.co/t/strange-split-brain-scenario/5782/3 "2011-11-14T10:34:43Z")

</div>

The relevant parts of the logs from es1-02 are here:  
[http://pastebin.com/aZshxLqn](http://pastebin.com/aZshxLqn)  
and the parts from es1-01 are here: [http://pastebin.com/et1MWMDG](http://pastebin.com/et1MWMDG)

Note that the reset we initiated was on Nov. 2nd at 4:24am. I can't be sure  
but we don't recall resetting the service on Nov. 1st at 00:21 when the log  
of es1-01 indicates a reset, also, the logs doesn't show a "stopping"  
message before those lines. Does ES has some built in watchdog that could  
do this?

Let me know if there is any additional information I can provide.

Thanks.

-eran

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 3:48am UTC](https://discuss.elastic.co/t/strange-split-brain-scenario/5782/4 "2017-07-06T03:48:49Z")

</div>

Closed on July 6, 2017, 3:48am UTC