# Master node can not rejoin cluster after restart on es2.0.0. but other nodes can rejoin cluster after restart #15916 **URL:** \ **Category:** Elasticsearch\ **Created:** [January 12, 2016, 7:06am UTC](https://discuss.elastic.co/t/master-node-can-not-rejoin-cluster-after-restart-on-es2-0-0-but-other-nodes-can-rejoin-cluster-after-restart-15916/38945 "2016-01-12T07:06:39Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![wisre](https://avatars.discourse-cdn.com/v4/letter/w/b3f665/32.png) [@wisre](https://discuss.elastic.co/u/wisre)\ **Post date:** [January 12, 2016, 7:06am UTC](https://discuss.elastic.co/t/master-node-can-not-rejoin-cluster-after-restart-on-es2-0-0-but-other-nodes-can-rejoin-cluster-after-restart-15916/38945/1 "2016-01-12T07:06:39Z")
i have a es cluster with ten nodes five dedicated master[A,B,C,D,E] and five datanode,when i retsart the master A ,other dedicated master elect another master B,but when the A start,it cannot find master B.but B can find A,and put A into the cluster. the error like this [2016-01-12 11:39:45,812][DEBUG][action.admin.cluster.state] [efe-es-03] no known master node, scheduling a retry [2016-01-12 11:39:45,891][DEBUG][action.admin.cluster.state] [efe-es-03] no known master node, scheduling a retry [2016-01-12 11:39:46,490][INFO][rest.suppressed] /.kibana/config/\_search Params: {index=.kibana, type=config} ClusterBlockException[blocked by: [SERVICE\_UNAVAILABLE/1/state not recovered / initialized];] at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:154) at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:144) the new master add A into cluster [2016-01-12 10:57:49,394][INFO][cluster.service] [efe-es-03] added {{efe-es-01}{9H7bnUbASVerOMdEOlWikg}{10.39.128.30}{10.39.128.30:9300}{rack=10.39.128.30, max\_local\_storage\_nodes=1, master=true},}, reason: zen-disco-join(join from node[{efe-es-01}{9H7bnUbASVerOMdEOlWikg}{10.39.128.30}{10.39.128.30:9300}{rack=10.39.128.30, max\_local\_storage\_nodes=1, master=true}]) but A can't get nodes info others can get nodes info including A curl localhost:8080/\_cat/nodes ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [January 12, 2016, 7:34am UTC](https://discuss.elastic.co/t/master-node-can-not-rejoin-cluster-after-restart-on-es2-0-0-but-other-nodes-can-rejoin-cluster-after-restart-15916/38945/2 "2016-01-12T07:34:03Z")
Logs are in a strange order. I think you did not copy all logs. Did you run the upgrade plugin first? May be you have to remove .kibana indices? ---
**Author:** ![wisre](https://avatars.discourse-cdn.com/v4/letter/w/b3f665/32.png) [@wisre](https://discuss.elastic.co/u/wisre)\ **Post date:** [January 12, 2016, 8:00am UTC](https://discuss.elastic.co/t/master-node-can-not-rejoin-cluster-after-restart-on-es2-0-0-but-other-nodes-can-rejoin-cluster-after-restart-15916/38945/3 "2016-01-12T08:00:49Z")
no i just want to roll restart node to add some params,when i restart other nodes which are not master,they can rejoin cluster well. but afteri restart master,then it gone bad,it can't rejoin master. i have delete all data on the bad node, but i can't join too. my cluster work well without the bad node ---
**Author:** ![wisre](https://avatars.discourse-cdn.com/v4/letter/w/b3f665/32.png) [@wisre](https://discuss.elastic.co/u/wisre)\ **Post date:** [January 12, 2016, 8:26am UTC](https://discuss.elastic.co/t/master-node-can-not-rejoin-cluster-after-restart-on-es2-0-0-but-other-nodes-can-rejoin-cluster-after-restart-15916/38945/4 "2016-01-12T08:26:44Z")
after i restart all the node in my cluster,i hava this error,the cluster is instability [2016-01-12 16:23:48,942][INFO][cluster.service] [bjx-efe-es-07] master {new {efe-es-01}{1bVuHy8NQv-P1yQPSt2AMg}{10.39.128.30}{10.39.128.30:9300}{rack=10.39.128.30, max\_local\_storage\_nodes=1, master=true}}, removed {{bjx-efe-es-06}{6EhE00oTQ2u9jVi1HtcNFQ}{10.39.128.39}{10.39.128.39:9300}{rack=10.39.128.39, max\_local\_storage\_nodes=1, master=true},{bjx-efe-es-08}{54boIymMQgyzJPo69TTIcw}{10.39.128.41}{10.39.128.41:9300}{rack=10.39.128.41, max\_local\_storage\_nodes=1, master=true},}, added {{efe-es-01}{1bVuHy8NQv-P1yQPSt2AMg}{10.39.128.30}{10.39.128.30:9300}{rack=10.39.128.30, max\_local\_storage\_nodes=1, master=true},}, reason: zen-disco-receive(from master [{efe-es-01}{1bVuHy8NQv-P1yQPSt2AMg}{10.39.128.30}{10.39.128.30:9300}{rack=10.39.128.30, max\_local\_storage\_nodes=1, master=true}]) [2016-01-12 16:23:49,375][INFO][cluster.service] [bjx-efe-es-07] removed {{efe-es-02}{v\_gaTkDySXWO4AI2jeL\_nw}{10.39.129.27}{10.39.129.27:9300}{rack=10.39.129.27, max\_local\_storage\_nodes=1, master=true},}, reason: zen-disco-receive(from master [{efe-es-01}{1bVuHy8NQv-P1yQPSt2AMg}{10.39.128.30}{10.39.128.30:9300}{rack=10.39.128.30, max\_local\_storage\_nodes=1, master=true}]) [2016-01-12 16:23:49,394][INFO][cluster.service] [bjx-efe-es-07] removed {{bjx-efe-es-10}{Oa5F3NP-TTCC9El2KUPnrQ}{10.39.129.56}{10.39.129.56:9300}{rack=10.39.129.56, max\_local\_storage\_nodes=1, master=true},}, reason: zen-disco-receive(from master [{efe-es-01}{1bVuHy8NQv-P1yQPSt2AMg}{10.39.128.30}{10.39.128.30:9300}{rack=10.39.128.30, max\_local\_storage\_nodes=1, master=true}]) [2016-01-12 16:23:49,944][INFO][discovery.zen] [bjx-efe-es-07] master\_left [{efe-es-01}{1bVuHy8NQv-P1yQPSt2AMg}{10.39.128.30}{10.39.128.30:9300}{rack=10.39.128.30, max\_local\_storage\_nodes=1, master=true}], reason [failed to ping, tried [3] times, each with maximum [30s] timeout] [2016-01-12 16:23:49,944][WARN][discovery.zen] [bjx-efe-es-07] master left (reason = failed to ping, tried [3] times, each with maximum [30s] timeout), current nodes: {{bjx-efe-es-09}{jwR1VIjIRymvxp2hSip8eA}{10.39.129.55}{10.39.129.55:9300}{rack=10.39.129.55, max\_local\_storage\_nodes=1, master=true},{efe-es-03}{6fV5VUH3Qp22LR6pp09vag}{10.39.129.28}{10.39.129.28:9300}{rack=10.39.129.28, max\_local\_storage\_nodes=1, master=true},{bjx-efe-es-07}{vCJnRZ1-Tw2g78I3IsWg5w}{10.39.128.40}{10.39.128.40:9300}{rack=10.39.128.40, max\_local\_storage\_nodes=1, master=true},{efe-es-05}{TdAANxmnRjeUWJk--k5ATQ}{10.39.129.30}{10.39.129.30:9300}{rack=10.39.129.30, max\_local\_storage\_nodes=1, master=true},{efe-es-04}{R8XrJ7jgRA6sU7q8LHBqmQ}{10.39.129.29}{10.39.129.29:9300}{rack=10.39.129.29, max\_local\_storage\_nodes=1, master=true},} [2016-01-12 16:23:49,945][INFO][cluster.service] [bjx-efe-es-07] removed {{efe-es-01}{1bVuHy8NQv-P1yQPSt2AMg}{10.39.128.30}{10.39.128.30:9300}{rack=10.39.128.30, max\_local\_storage\_nodes=1, master=true},}, reason: zen-disco-master\_failed ({efe-es-01}{1bVuHy8NQv-P1yQPSt2AMg}{10.39.128.30}{10.39.128.30:9300}{rack=10.39.128.30, max\_local\_storage\_nodes=1, master=true}) [2016-01-12 16:23:49,946][INFO][discovery.zen] [bjx-efe-es-07] failed to send join request to master [{efe-es-01}{1bVuHy8NQv-P1yQPSt2AMg}{10.39.128.30}{10.39.128.30:9300}{rack=10.39.128.30, max\_local\_storage\_nodes=1, master=true}], reason [NodeDisconnectedException[[efe-es-01][10.39.128.30:9300][internal:discovery/zen/join] disconnected]] [2016-01-12 16:23:52,963][INFO][cluster.service] [bjx-efe-es-07] detected\_master {efe-es-01}{1bVuHy8NQv-P1yQPSt2AMg}{10.39.128.30}{10.39.128.30:9300}{rack=10.39.128.30, max\_local\_storage\_nodes=1, master=true}, added {{bjx-efe-es-08}{54boIymMQgyzJPo69TTIcw}{10.39.128.41}{10.39.128.41:9300}{rack=10.39.128.41, max\_local\_storage\_nodes=1, master=true},{efe-es-01}{1bVuHy8NQv-P1yQPSt2AMg}{10.39.128.30}{10.39.128.30:9300}{rack=10.39.128.30, max\_local\_storage\_nodes=1, master=true},{efe-es-02}{v\_gaTkDySXWO4AI2jeL\_nw}{10.39.129.27}{10.39.129.27:9300}{rack=10.39.129.27, max\_local\_storage\_nodes=1, master=true},{bjx-efe-es-06}{6EhE00oTQ2u9jVi1HtcNFQ}{10.39.128.39}{10.39.128.39:9300}{rack=10.39.128.39, max\_local\_storage\_nodes=1, master=true},}, reason: zen-disco-receive(from master [{efe-es-01}{1bVuHy8NQv-P1yQPSt2AMg}{10.39.128.30}{10.39.128.30:9300}{rack=10.39.128.30, max\_local\_storage\_nodes=1, master=true}]) [2016-01-12 16:23:52,972][INFO][cluster.service] [bjx-efe-es-07] added {{bjx-efe-es-10}{Oa5F3NP-TTCC9El2KUPnrQ}{10.39.129.56}{10.39.129.56:9300}{rack=10.39.129.56, max\_local\_storage\_nodes=1, [2016-01-12 16:23:54,966][INFO][discovery.zen] [bjx-efe-es-07] master\_left [{efe-es-01}{1bVuHy8NQv-P1yQPSt2AMg}{10.39.128.30}{10.39.128.30:9300}{rack=10.39.128.30, max\_local\_storage\_nodes=1, master=true}], ---
**Author:** ![wisre](https://avatars.discourse-cdn.com/v4/letter/w/b3f665/32.png) [@wisre](https://discuss.elastic.co/u/wisre)\ **Post date:** [January 12, 2016, 9:41am UTC](https://discuss.elastic.co/t/master-node-can-not-rejoin-cluster-after-restart-on-es2-0-0-but-other-nodes-can-rejoin-cluster-after-restart-15916/38945/5 "2016-01-12T09:41:54Z")
the all log like this,not discover master [2016-01-12 17:38:47,175][INFO][rest.action.readonlyrest] [efe-es-03] Readonly REST plugin was loaded... [2016-01-12 17:38:47,175][INFO][rest.action.readonlyrest] [efe-es-03] Readonly Rest plugin is installed, but not enabled [2016-01-12 17:38:47,175][INFO][rest.action.readonlyrest] [efe-es-03] Readonly REST plugin is disabled! [2016-01-12 17:38:47,255][INFO][node] [efe-es-03] initialized [2016-01-12 17:38:47,255][INFO][node] [efe-es-03] starting ... [2016-01-12 17:38:47,432][INFO][transport] [efe-es-03] publish\_address {10.39.129.28:9300}, bound\_addresses {10.39.129.28:9300} [2016-01-12 17:38:47,440][INFO][discovery] [efe-es-03] elongELK/opB8sMAWTuOGld-qtUbiAQ [2016-01-12 17:39:17,440][WARN][discovery] [efe-es-03] waited for 30s and no initial state was set by the discovery [2016-01-12 17:39:17,502][INFO][http] [efe-es-03] publish\_address {10.39.129.28:8080}, bound\_addresses {10.39.129.28:8080} [2016-01-12 17:39:17,503][INFO][node] [efe-es-03] started [2016-01-12 17:39:22,763][INFO][rest.suppressed] /.kibana/config/\_search Params: {index=.kibana, type=config} ClusterBlockException[blocked by: [SERVICE\_UNAVAILABLE/1/state not recovered / initialized];] at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:154) at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:144) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 11:25pm UTC](https://discuss.elastic.co/t/master-node-can-not-rejoin-cluster-after-restart-on-es2-0-0-but-other-nodes-can-rejoin-cluster-after-restart-15916/38945/6 "2017-07-05T23:25:14Z")