# Split brain, confused master: failed to send rejoin request to \ **URL:** https://discuss.elastic.co/t/split-brain-confused-master-failed-to-send-rejoin-request-to-host/9767 **Category:** Elasticsearch **Created:** [November 20, 2012, 7:24am UTC](https://discuss.elastic.co/t/split-brain-confused-master-failed-to-send-rejoin-request-to-host/9767 "2012-11-20T07:24:19Z") **Posts on this page:** 10 **Page:** 1
### Author: ![wvl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wvl/32/2609_2.png) [@wvl](https://discuss.elastic.co/u/wvl) #### Post date: [November 20, 2012, 7:24am UTC](https://discuss.elastic.co/t/split-brain-confused-master-failed-to-send-rejoin-request-to-host/9767/1 "2012-11-20T07:24:19Z")
Nothing showed up on google, so here we go: Currently 3 nodes don't want to join the cluster of 15+ nodes. one of the nodes logs: [2012-11-20 07:17:35,623][DEBUG][discovery.zen.fd] [es-028] [master] starting fault detection against master [[es-030][CgNqNgzcRGOFan4QJY6gGg][inet[/10.32.0.137:29300]]{name\_attr=es-030, master=true, river=_none_}], reason [initial\_join] [2012-11-20 07:17:36,650][DEBUG][discovery.zen.fd] [es-028] [master] pinging a master [es-030][CgNqNgzcRGOFan4QJY6gGg][inet[/10.32.0.137:29300]]{name\_attr=es-030, master=true, river=_none_} but we do not exists on it, act as if its master failure [2012-11-20 07:17:36,651][DEBUG][discovery.zen.fd] [es-028] [master] stopping fault detection against master [[es-030][CgNqNgzcRGOFan4QJY6gGg][inet[/10.32.0.137:29300]]{name\_attr=es-030, master=true, river=_none_}], reason [master failure, do not exists on master, act as master failure] [2012-11-20 07:17:36,652][INFO][discovery.zen] [es-028] master\_left [[es-030][CgNqNgzcRGOFan4QJY6gGg][inet[/10.32.0.137:29300]]{name\_attr=es-030, master=true, river=_none_}], reason [do not exists on master, act as master failure] master reports: [2012-11-20 07:16:35,257][TRACE][discovery.zen.ping.multicast] [es-030] [1] received ping\_request from [[es-028][33TXhg8vTR6FrLdlwjwMvw][inet[/10.32.0.135:29300]]{name\_attr=es-028, master=true, river=_none_}], sending ping\_response{target [[es-030][CgNqNgzcRGOFan4QJY6gGg][inet[/10.32.0.137:29300]]{name\_attr=es-030, master=true, river=_none_}], master [[es-030][CgNqNgzcRGOFan4QJY6gGg][inet[/10.32.0.137:29300]]{name\_attr=es-030, master=true, river=_none_}], cluster\_name[bc]} [2012-11-20 07:17:05,259][TRACE][discovery.zen.ping.multicast] [es-030] [1] received ping\_request from [[es-028][33TXhg8vTR6FrLdlwjwMvw][inet[/10.32.0.135:29300]]{name\_attr=es-028, master=true, river=_none_}], sending ping\_response{target [[es-030][CgNqNgzcRGOFan4QJY6gGg][inet[/10.32.0.137:29300]]{name\_attr=es-030, master=true, river=_none_}], master [[es-030][CgNqNgzcRGOFan4QJY6gGg][inet[/10.32.0.137:29300]]{name\_attr=es-030, master=true, river=_none_}], cluster\_name[bc]} No network issues.. telneting from port to port works fine. ## Config from one of the nodes, removed filters and stuff: cluster: name: bc node: name: ${SHORT\_HOSTNAME} name\_attr: ${SHORT\_HOSTNAME} river: "_none_" master: true index: number\_of\_shards: 29 number\_of\_replicas: 3 gateway.type: local gateway.recover\_after\_nodes: 9 gateway.recover\_after\_time: 5m gateway.expected\_nodes: 10 cluster.routing.allocation.node\_initial\_primaries\_recoveries: 10 cluster.routing.allocation.node\_concurrent\_recoveries: 5 cluster.routing.allocation.cluster\_concurrent\_rebalance: 20 discovery.zen.minimum\_master\_nodes: 3 discovery.zen.ping.timeout: 60s discovery.zen.fd.ping\_timeout: 60s transport.tcp.port: 29300 http.port: 29200 index.search.slowlog.level: TRACE index.search.slowlog.threshold.query.warn: 60s index.search.slowlog.threshold.query.info: 10s index.search.slowlog.threshold.query.debug: 5s index.search.slowlog.threshold.query.trace: 2s index.search.slowlog.threshold.fetch.warn: 1s index.search.slowlog.threshold.fetch.info: 800ms index.search.slowlog.threshold.fetch.debug: 500ms index.search.slowlog.threshold.fetch.trace: 200ms * * * already tried restarting the rogue nodes to no avail. Any help would be appreciated! -- ---
### Author: ![drewr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/drewr/32/7803_2.png) [@drewr](https://discuss.elastic.co/u/drewr) #### Post date: [November 20, 2012, 3:19pm UTC](https://discuss.elastic.co/t/split-brain-confused-master-failed-to-send-rejoin-request-to-host/9767/2 "2012-11-20T15:19:55Z")
wvl wrote: > Nothing showed up on google, so here we go: > > Currently 3 nodes don't want to join the cluster of 15+ nodes. [...] > already tried restarting the rogue nodes to no avail. Any help would be > appreciated! If you haven't done it already, stop all indexing! We need to inspect the split brain situation more closely. You need to ask each node who it thinks is master. Use parallel-ssh, knife, or whatever else allows you to run a command across all of the nodes. Here's an example using pssh/jsawk (you could use any tool you like) from [https://gist.github.com/4118487:](https://gist.github.com/4118487:) pssh es curl -s [https://raw.github.com/gist/4118487/master.sh](https://raw.github.com/gist/4118487/master.sh) | sh If you use that, change the variables to reflect your network interface and ES HTTP port. Output will look something like this for each node in the cluster unless. If you're running more than one node on each machine you'll have to tune it even further. me: 192.168.20.109 master: ["bYijI5\_vT3CsrhdukyCUMA","Lifeforce","inet[/192.168.20.109:9300]"] That will give us a sense of how many different masters there are and how the nodes disagree. Then we can examine logs in more detail and figure out what to do. -Drew -- ---
### Author: ![wvl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wvl/32/2609_2.png) [@wvl](https://discuss.elastic.co/u/wvl) #### Post date: [November 21, 2012, 12:12am UTC](https://discuss.elastic.co/t/split-brain-confused-master-failed-to-send-rejoin-request-to-host/9767/3 "2012-11-21T00:12:28Z")
> If you haven't done it already, stop all indexing! When the cluster hits status red it already stops indexing right? > We need to inspect the split brain situation more closely Right, did so at the time. Saw that we had 2 clusters: one of 3 nodes and another of lets say 10+. I provided log entries of the master with the most nodes and one of the rogue nodes that refused to join the master's cluster, even if we shutdown down all other nodes in the smaller cluster and make it so that this node can become master itself. So basically whatever we did we couldnt get the rogue to join the larger cluster, with the only relevant logs being provided above. -- ---
### Author: ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan) #### Post date: [November 21, 2012, 12:27am UTC](https://discuss.elastic.co/t/split-brain-confused-master-failed-to-send-rejoin-request-to-host/9767/4 "2012-11-21T00:27:00Z")
The minimum\_master\_nodes settings seems eerily identical to the size of the rogue cluster. Coincidence? I think not! Ideally this settings should be (n/2)+1 where n is the number of nodes. Not sure of the significance of expected\_nodes, but I set it to the number of nodes. Can't say why the cluster lost communication in the first place, but you can avoid split brains. I have played around with the transport timeout setting because nodes sometimes fail to communicate under high load scenarios. Cheers, Ivan On Mon, Nov 19, 2012 at 11:24 PM, wvl [william.leese@meltwater.com](mailto:william.leese@meltwater.com) wrote: > Currently 3 nodes don't want to join the cluster of 15+ nodes. > > gateway.expected\_nodes: 10 > discovery.zen.minimum\_master\_nodes: 3 -- ---
### Author: ![wvl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wvl/32/2609_2.png) [@wvl](https://discuss.elastic.co/u/wvl) #### Post date: [November 21, 2012, 2:36am UTC](https://discuss.elastic.co/t/split-brain-confused-master-failed-to-send-rejoin-request-to-host/9767/5 "2012-11-21T02:36:36Z")
Right, we agree on the minimum\_master\_nodes. > Can't say why the cluster lost communication in the first place So this is the issue that really concerns me - these unexplained node failures. -- ---
### Author: ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov) #### Post date: [November 21, 2012, 2:50pm UTC](https://discuss.elastic.co/t/split-brain-confused-master-failed-to-send-rejoin-request-to-host/9767/6 "2012-11-21T14:50:48Z")
Node failures typically occur when nodes don't respond to pings in timely manner. There are two primary reasons for ping failures - network issues and node being busy. A node can get unresponsive for a number of reasons. For example, if you are not using mlockall option, a part of elasticsearch memory can get swapped out and when GC kicks in on such node, it can render node unresponsive for a long time if there is not enough physical memory on the system. So, memory would be the first thing to check. What's the max heap size for your nodes and how much memory your servers have? Another possible reason for unresponsive nodes is too much traffic. Are you monitoring CPU and I/O load on your elasticsearch nodes? How were nodes doing just before the failures occurred? On Tuesday, November 20, 2012 9:36:36 PM UTC-5, wvl wrote: > Right, we agree on the minimum\_master\_nodes. > > > Can't say why the cluster lost communication in the first place > > So this is the issue that really concerns me - these unexplained node > failures. -- ---
### Author: ![wvl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wvl/32/2609_2.png) [@wvl](https://discuss.elastic.co/u/wvl) #### Post date: [November 25, 2012, 11:22pm UTC](https://discuss.elastic.co/t/split-brain-confused-master-failed-to-send-rejoin-request-to-host/9767/7 "2012-11-25T23:22:40Z")
Hi Igor, Thanks for joining the discussion. We have all these monitors in place and saw nothing alarming (IO, RAM, CPU-wise), except a rise in the field\_cache. We suspect there might be a problem there especially since we had similiar issues with the field cache growing out of hand in the past. Regardless, this didnt cause any OOM condition or any obvious (CPU,RAM,IO) resource problem. On Wednesday, November 21, 2012 11:50:48 PM UTC+9, Igor Motov wrote: > Node failures typically occur when nodes don't respond to pings in timely > manner. There are two primary reasons for ping failures - network issues > and node being busy. A node can get unresponsive for a number of reasons. > For example, if you are not using mlockall option, a part of elasticsearch > memory can get swapped out and when GC kicks in on such node, it can render > node unresponsive for a long time if there is not enough physical memory on > the system. So, memory would be the first thing to check. What's the max > heap size for your nodes and how much memory your servers have? Another > possible reason for unresponsive nodes is too much traffic. Are you > monitoring CPU and I/O load on your elasticsearch nodes? How were nodes > doing just before the failures occurred? > > On Tuesday, November 20, 2012 9:36:36 PM UTC-5, wvl wrote: > > > Right, we agree on the minimum\_master\_nodes. > > > > > Can't say why the cluster lost communication in the first place > > > > So this is the issue that really concerns me - these unexplained node > > failures. -- ---
### Author: ![Karussell\_2](https://avatars.discourse-cdn.com/v4/letter/k/54ee81/32.png) [@Karussell\_2](https://discuss.elastic.co/u/Karussell_2) #### Post date: [November 26, 2012, 3:10pm UTC](https://discuss.elastic.co/t/split-brain-confused-master-failed-to-send-rejoin-request-to-host/9767/8 "2012-11-26T15:10:51Z")
On Wednesday, November 21, 2012 1:12:28 AM UTC+1, wvl wrote: > > If you haven't done it already, stop all indexing! > > When the cluster hits status red it already stops indexing right? No! Have a look into this nice demonstration from Lukas and Karel: [http://vimeo.com/44718093](http://vimeo.com/44718093) (12:00) Regards, Peter. -- ---
### Author: ![Chuck\_McKenzie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chuck_mckenzie/32/19660_2.png) [@Chuck\_McKenzie](https://discuss.elastic.co/u/Chuck_McKenzie) #### Post date: [November 26, 2012, 11:15pm UTC](https://discuss.elastic.co/t/split-brain-confused-master-failed-to-send-rejoin-request-to-host/9767/9 "2012-11-26T23:15:45Z")
I've started to use the following in nagios to watch for load-induced split brain. It queries a node, finds all the nodes in its cluster, and makes sure they all have the same master\_node. My environment has node.name set to hostname for readability, so it may not work out of the box if yours doesn't. > **[GitHub - redchuck/es-check-splitbrain: Checks all nodes in an elasticsearch...](https://github.com/redchuck/es-check-splitbrain)** > > Checks all nodes in an elasticsearch cluster to make sure they have the same master. - GitHub - redchuck/es-check-splitbrain: Checks all nodes in an elasticsearch cluster to make sure they have the... On Tuesday, November 20, 2012 9:19:40 AM UTC-6, Drew Raines wrote: > wvl wrote: > > > Nothing showed up on google, so here we go: > > > > Currently 3 nodes don't want to join the cluster of 15+ nodes. > > [...] > > > already tried restarting the rogue nodes to no avail. Any help would be > > appreciated! > > If you haven't done it already, stop all indexing! > > We need to inspect the split brain situation more closely. You need > to ask each node who it thinks is master. Use parallel-ssh, knife, > or whatever else allows you to run a command across all of the nodes. > Here's an example using pssh/jsawk (you could use any tool you like) > from [https://gist.github.com/4118487:](https://gist.github.com/4118487:) > > pssh es curl -s [https://raw.github.com/gist/4118487/master.sh](https://raw.github.com/gist/4118487/master.sh) | sh > > If you use that, change the variables to reflect your network > interface and ES HTTP port. Output will look something like this for > each node in the cluster unless. If you're running more than one > node on each machine you'll have to tune it even further. > > me: 192.168.20.109 master: ["bYijI5\_vT3CsrhdukyCUMA","Lifeforce","inet[/ > 192.168.20.109:9300]"] > > That will give us a sense of how many different masters there are and > how the nodes disagree. Then we can examine logs in more detail and > figure out what to do. > > -Drew -- ---
### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system) #### Post date: [July 6, 2017, 3:02am UTC](https://discuss.elastic.co/t/split-brain-confused-master-failed-to-send-rejoin-request-to-host/9767/10 "2017-07-06T03:02:54Z")