# Master election problem in 3 node cluster when one died **URL:** \ **Category:** Elasticsearch\ **Created:** [March 3, 2016, 7:56pm UTC](https://discuss.elastic.co/t/master-election-problem-in-3-node-cluster-when-one-died/43433 "2016-03-03T19:56:46Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![ehogan](https://avatars.discourse-cdn.com/v4/letter/e/b487fb/32.png) [@ehogan](https://discuss.elastic.co/u/ehogan)\ **Post date:** [March 3, 2016, 7:56pm UTC](https://discuss.elastic.co/t/master-election-problem-in-3-node-cluster-when-one-died/43433/1 "2016-03-03T19:56:46Z")
I have a pretty simple ELK stack here with 4 ES nodes (3 data nodes and 1 client node). I lost one of the data nodes and all of a sudden my Logstash servers couldn't send anything to the ES cluster. I found this error on the LS nodes and on the surviving ES nodes: {"error": {"root\_cause": [{"type":"cluster\_block\_exception", "reason":"blocked by: [SERVICE\_UNAVAILABLE/2/no master];"}], "type":"cluster\_block\_exception", "reason":"blocked by: [SERVICE\_UNAVAILABLE/2/no master];"},"status":503}", :class=\>"Elasticsearch::Transport::Transport::Errors::ServiceUnavailable", .... I am assuming that this is because the ES cluster could no longer elect a master...but I thought that 2 nodes in a 3 node cluster were enough. Did I miss something along the way? (Sorry if this is basic ES knowledge. I found a posting about someone else running into this problem as well during a rolling upgrade, but there was not a response.) I am running ES version 2.2.0. Thanks in advance. -Emmett ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 3, 2016, 7:59pm UTC](https://discuss.elastic.co/t/master-election-problem-in-3-node-cluster-when-one-died/43433/2 "2016-03-03T19:59:51Z")
What is minimum\_master\_nodes set to? ---
**Author:** ![ehogan](https://avatars.discourse-cdn.com/v4/letter/e/b487fb/32.png) [@ehogan](https://discuss.elastic.co/u/ehogan)\ **Post date:** [March 3, 2016, 8:15pm UTC](https://discuss.elastic.co/t/master-election-problem-in-3-node-cluster-when-one-died/43433/3 "2016-03-03T20:15:22Z")
Uhhh....crap. I thought that it was automatically computed to (# nodes/2 + 1) if not defined...but now that I read the config...that's not exactly what it says. So...it's actually commented out in my config! Doh! So...I should have: discovery.zen.minimum\_master\_nodes: 2 I am guessing that the _right_ way to update this would be: - Change it on all three nodes - Shut down all my logstash nodes so nothing is getting sent to ES. - Shut down each ES node - Start each ES node Otherwise, I'll run into the same problem as soon as I restart ES on a node to reload the new config. Right? Thanks for your help! -Emmett ---
**Author:** ![ehogan](https://avatars.discourse-cdn.com/v4/letter/e/b487fb/32.png) [@ehogan](https://discuss.elastic.co/u/ehogan)\ **Post date:** [March 3, 2016, 8:25pm UTC](https://discuss.elastic.co/t/master-election-problem-in-3-node-cluster-when-one-died/43433/4 "2016-03-03T20:25:50Z")
While I am changing things in my config...should I also set: gateway.recover\_after\_nodes: 2 -Emmett ---
**Author:** ![ehogan](https://avatars.discourse-cdn.com/v4/letter/e/b487fb/32.png) [@ehogan](https://discuss.elastic.co/u/ehogan)\ **Post date:** [March 3, 2016, 8:44pm UTC](https://discuss.elastic.co/t/master-election-problem-in-3-node-cluster-when-one-died/43433/5 "2016-03-03T20:44:09Z")
Answering my own question...I found this... [https://www.elastic.co/guide/en/elasticsearch/reference/current/restart-upgrade.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/restart-upgrade.html) -E ---
**Author:** ![ehogan](https://avatars.discourse-cdn.com/v4/letter/e/b487fb/32.png) [@ehogan](https://discuss.elastic.co/u/ehogan)\ **Post date:** [March 3, 2016, 9:47pm UTC](https://discuss.elastic.co/t/master-election-problem-in-3-node-cluster-when-one-died/43433/6 "2016-03-03T21:47:43Z")
I just noticed something strange though. I follow the right procedure for restarting my cluster: 1. Turned off shard reallocation 2. Bounced the nodes 3. Turned on shard reallocation and everything looked fine...except that the last node that I brought up only has replicas on it. No primary shards at all! -Emmett ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 4, 2016, 1:30am UTC](https://discuss.elastic.co/t/master-election-problem-in-3-node-cluster-when-one-died/43433/7 "2016-03-04T01:30:44Z")
That's nothing to be worried about 🙂 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 11:11pm UTC](https://discuss.elastic.co/t/master-election-problem-in-3-node-cluster-when-one-died/43433/8 "2017-07-05T23:11:24Z")