# Cluster Split Brain **URL:** \ **Category:** Elasticsearch\ **Created:** [September 13, 2011, 6:43pm UTC](https://discuss.elastic.co/t/cluster-split-brain/5373 "2011-09-13T18:43:46Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![phobos182](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/phobos182/32/3011_2.png) [@phobos182](https://discuss.elastic.co/u/phobos182)\ **Post date:** [September 13, 2011, 6:43pm UTC](https://discuss.elastic.co/t/cluster-split-brain/5373/1 "2011-09-13T18:43:46Z")
We have been having issues with our cluster split braining a few times this week. Here is the error logs. Anything I can take a look at? [http://www.pastie.org/2527719](http://www.pastie.org/2527719) [http://www.pastie.org/pastes/2527779](http://www.pastie.org/pastes/2527779) ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [September 13, 2011, 8:12pm UTC](https://discuss.elastic.co/t/cluster-split-brain/5373/2 "2011-09-13T20:12:36Z")
Can you check the logs, see why the nodes disconnect (you should see log messages on nodes being disconnected). On Tue, Sep 13, 2011 at 9:43 PM, phobos182 [phobos182@gmail.com](mailto:phobos182@gmail.com) wrote: > We have been having issues with our cluster split braining a few times this > week. Here is the error logs. Anything I can take a look at? > > [http://www.pastie.org/2527719](http://www.pastie.org/2527719) > > -- > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/Cluster-Split-Brain-tp3333510p3333510.html](http://elasticsearch-users.115913.n3.nabble.com/Cluster-Split-Brain-tp3333510p3333510.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![phobos182](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/phobos182/32/3011_2.png) [@phobos182](https://discuss.elastic.co/u/phobos182)\ **Post date:** [September 15, 2011, 3:04pm UTC](https://discuss.elastic.co/t/cluster-split-brain/5373/3 "2011-09-15T15:04:15Z")
It happened again, and this time I have the log files. ## 192.168.200.110 [http://www.pastie.org/pastes/2537779](http://www.pastie.org/pastes/2537779) ## 192.168.200.109 [http://www.pastie.org/pastes/2537721](http://www.pastie.org/pastes/2537721) ## Master [http://www.pastie.org/pastes/2537751](http://www.pastie.org/pastes/2537751) [http://www.pastie.org/pastes/2537894](http://www.pastie.org/pastes/2537894) It looks like the "Master" went offline where it could not ping anybody else, or something to that fact. But it came back online shortly. Why would the cluster split brain if one node went down (That happens to be the master)? I'm sure the cluster had quorum to route around a failed node. It also looks like the master went haywire with the cluster updates during that time. Went from version 900 to 925 in less than a second. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [September 16, 2011, 6:49pm UTC](https://discuss.elastic.co/t/cluster-split-brain/5373/4 "2011-09-16T18:49:22Z")
I don't see the log messages of the node being disconnected from the cluster. On Thu, Sep 15, 2011 at 6:04 PM, phobos182 [phobos182@gmail.com](mailto:phobos182@gmail.com) wrote: > It happened again, and this time I have the log files. > > [http://www.pastie.org/pastes/2537721](http://www.pastie.org/pastes/2537721) > > -- > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/Cluster-Split-Brain-tp3333510p3339227.html](http://elasticsearch-users.115913.n3.nabble.com/Cluster-Split-Brain-tp3333510p3339227.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![phobos182](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/phobos182/32/3011_2.png) [@phobos182](https://discuss.elastic.co/u/phobos182)\ **Post date:** [September 22, 2011, 12:45pm UTC](https://discuss.elastic.co/t/cluster-split-brain/5373/5 "2011-09-22T12:45:00Z")
I wanted to follow up on this. It turns out that the version of the CentOS 6 kernel that we were using had an issue with the bnx2 driver for the BroadCom NICs in our servers which would cause very short lived, but temporary outages for the interfaces. We updated the Kernel drivers, rebooted each node, and have not seen this error since. The tip off was the error message in the logs that a server could not be pinged for 20-30s. Considering we have a very robust 480Gbit/s network core with 10GB LAG's, this pointed directly to the servers. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:53am UTC](https://discuss.elastic.co/t/cluster-split-brain/5373/6 "2017-07-06T03:53:48Z")