# Elasticsearch 3 node cluster failing if master is down second time **URL:** \ **Category:** Elasticsearch\ **Created:** [May 27, 2017, 8:09pm UTC](https://discuss.elastic.co/t/elasticsearch-3-node-cluster-failing-if-master-is-down-second-time/87348 "2017-05-27T20:09:56Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![samdash](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samdash/32/21628_2.png) [@samdash](https://discuss.elastic.co/u/samdash)\ **Post date:** [May 27, 2017, 8:09pm UTC](https://discuss.elastic.co/t/elasticsearch-3-node-cluster-failing-if-master-is-down-second-time/87348/1 "2017-05-27T20:09:56Z")
I have a 3 node cluster and wanted to know the best possible configuration of elastic search. Below is the configuration I have tried. **Server 1,2,3** ``` node.name : node1 [node2, node3] node.master: true node.data: true ``` it works fine, however while testing It is failing in one scenario.Below are the tests i have performed. **Test 1** - Bring All 3 nodes up and shutdown master node in this case **node1** is mater , Cluster state goes to yellow and back to Green after a while with 2 modes running. - Now bring back the node that was shutdown , now all 3 nodes were up and now **node2** is elected as master and everything is working fine **Test2** - After Test 1 is run and all 3 nodes were up since **node2** is elected as master, I have shutdown **node2** , now i am not able to connect and get failed to connect error below **curl: (7) Failed to connect to localhost port 9200: Connection refused** - However **node1** which is elected as master show's below message with Cluster health status changed from [YELLOW] to [GREEN] **Message** ``` [2017-05-26T22:52:23,996][INFO][o.e.c.r.a.AllocationService] [node-1] Cluster health status changed from [GREEN] to [YELLOW] (reason: [{node-2}{6THtjHP4SuKsrAZYcOv6Sw}{wyC6CjSESOOisEMTOE_wCw}{127.0.0.1}{127.0.0.1:9301} transport disconnected, {node-2}{6THtjHP4SuKsrAZYcOv6Sw}{wyC6CjSESOOisEMTOE_wCw}{127.0.0.1}{127.0.0.1:9301} transport disconnected]). [2017-05-26T22:52:23,997][INFO][o.e.c.s.ClusterService] [node-1] removed {{node-2}{6THtjHP4SuKsrAZYcOv6Sw}{wyC6CjSESOOisEMTOE_wCw}{127.0.0.1}{127.0.0.1:9301},}, reason: zen-disco-node-failed({node-2}{6THtjHP4SuKsrAZYcOv6Sw} {wyC6CjSESOOisEMTOE_wCw}{127.0.0.1}{127.0.0.1:9301}), reason(transport disconnected)[{node-2}{6THtjHP4SuKsrAZYcOv6Sw}{wyC6CjSESOOisEMTOE_wCw}{127.0.0.1} {127.0.0.1:9301} transport disconnected, {node-2}{6THtjHP4SuKsrAZYcOv6Sw}{wyC6CjSESOOisEMTOE_wCw}{127.0.0.1}{127.0.0.1:9301} transport disconnected] [2017-05-26T22:52:24,099][INFO][o.e.c.r.DelayedAllocationService] [node-1] scheduling reroute for delayed shards in [59.7s] (3 delayed shards) [2017-05-26T22:53:26,115][INFO][o.e.c.r.a.AllocationService] [node-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[shakespeare][3], [shakespeare][4]] ...]). ``` **Test3** ``` After Test 1 is run and all 3 nodes were up since **node2** is elected as master, I have shutdown **node3** , now everything is working fine since I did not shutdown master the second time ``` ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 27, 2017, 10:35pm UTC](https://discuss.elastic.co/t/elasticsearch-3-node-cluster-failing-if-master-is-down-second-time/87348/2 "2017-05-27T22:35:32Z")
This is expected behaviour given your configuration. However is there something you'd like to ask here, as it's not clear 🙂 ---
**Author:** ![samdash](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samdash/32/21628_2.png) [@samdash](https://discuss.elastic.co/u/samdash)\ **Post date:** [May 27, 2017, 10:42pm UTC](https://discuss.elastic.co/t/elasticsearch-3-node-cluster-failing-if-master-is-down-second-time/87348/3 "2017-05-27T22:42:11Z")
thanks for response warkolm, So there is no solution ? I am randomly bring down the master node , first time when i bring down it works fine, however when i bring down the second time, it fails. all my 3 nodes are master eligible node.master : true node.data : true not sure why cluster does not respond the second time , let me know what is the best configuration for 3 node cluster ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 27, 2017, 11:03pm UTC](https://discuss.elastic.co/t/elasticsearch-3-node-cluster-failing-if-master-is-down-second-time/87348/4 "2017-05-27T23:03:11Z")
> [@samdash](#): > > however when i bring down the second time, it fails. Are you talking about when you curl the node you stopped? That's entirely expected. ---
**Author:** ![samdash](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samdash/32/21628_2.png) [@samdash](https://discuss.elastic.co/u/samdash)\ **Post date:** [May 28, 2017, 1:02am UTC](https://discuss.elastic.co/t/elasticsearch-3-node-cluster-failing-if-master-is-down-second-time/87348/5 "2017-05-28T01:02:05Z")
> [@warkolm](#): > > Are you talking about when you curl the node you stopped? That's entirely expected. when i run curl command for cluster health , it is not working. curl -XGET 'localhost:9200/\_cluster/health?pretty' curl: (7) Failed to connect to localhost port 9200: Connection refused My elastic head , does not show any nodes , when i shut down the master second time only. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 28, 2017, 1:48am UTC](https://discuss.elastic.co/t/elasticsearch-3-node-cluster-failing-if-master-is-down-second-time/87348/6 "2017-05-28T01:48:58Z")
If you are on that node and then stop ES then of course it won't respond. You need to contact a different node. ---
**Author:** ![samdash](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samdash/32/21628_2.png) [@samdash](https://discuss.elastic.co/u/samdash)\ **Post date:** [May 28, 2017, 7:35pm UTC](https://discuss.elastic.co/t/elasticsearch-3-node-cluster-failing-if-master-is-down-second-time/87348/7 "2017-05-28T19:35:47Z")
Thanks, currently i am testing 3 nodes cluster on my mac ( single machine) . maybe that is the reason. anyhow , sometimes i am able to connect to the cluster sometimes not. will debug more. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 28, 2017, 8:46pm UTC](https://discuss.elastic.co/t/elasticsearch-3-node-cluster-failing-if-master-is-down-second-time/87348/8 "2017-05-28T20:46:57Z")
> [@samdash](#): > > currently i am testing 3 nodes cluster on my mac ( single machine) Right, well that makes more sense! > [@samdash](#): > > 127.0.0.1:9301 See how that is not 9300? When you start another node on the same host, ES picks 9300+1, if you start another it's 9301+2 etc etc. It's the same for the HTTP port on 9200. First node gets 9200, second 9201, third 9203. So if you stop the first node you need to `curl localhost:9201`. ---
**Author:** ![samdash](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samdash/32/21628_2.png) [@samdash](https://discuss.elastic.co/u/samdash)\ **Post date:** [May 28, 2017, 8:53pm UTC](https://discuss.elastic.co/t/elasticsearch-3-node-cluster-failing-if-master-is-down-second-time/87348/9 "2017-05-28T20:53:31Z")
Oh! Thanks a lot, it worked like you mentioned , Appreciate it a lot. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 25, 2017, 8:53pm UTC](https://discuss.elastic.co/t/elasticsearch-3-node-cluster-failing-if-master-is-down-second-time/87348/10 "2017-06-25T20:53:50Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.