# ES failure for few seconds during master re-elect **URL:** \ **Category:** Elasticsearch\ **Created:** [March 2, 2015, 5:16pm UTC](https://discuss.elastic.co/t/es-failure-for-few-seconds-during-master-re-elect/22472 "2015-03-02T17:16:46Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Ankush\_Jhalani](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ankush_jhalani/32/875_2.png) [@Ankush\_Jhalani](https://discuss.elastic.co/u/Ankush_Jhalani)\ **Post date:** [March 2, 2015, 5:16pm UTC](https://discuss.elastic.co/t/es-failure-for-few-seconds-during-master-re-elect/22472/1 "2015-03-02T17:16:46Z")
We have a 8 node cluster, with 1-master and 3-replica for each shard distributed across 2 datacenters (so 2 copies of each shard/per datacenter). When we shut-down a node for maintenance, ES takes few seconds to re-elect new master. During this time however ES requests on _other nodes_ fail with error - > {"error":"ClusterBlockException[blocked by: [SERVICE\_UNAVAILABLE/2/no > master];]","status":503} -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/46811878-c8d5-4c31-b10a-5c66b6ad1673%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/46811878-c8d5-4c31-b10a-5c66b6ad1673%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Ankush\_Jhalani](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ankush_jhalani/32/875_2.png) [@Ankush\_Jhalani](https://discuss.elastic.co/u/Ankush_Jhalani)\ **Post date:** [March 2, 2015, 5:17pm UTC](https://discuss.elastic.co/t/es-failure-for-few-seconds-during-master-re-elect/22472/2 "2015-03-02T17:17:50Z")
Logs on a node which failed ES requests - > [2015-02-25 04:03:31,701][DEBUG][transport.netty] [ny1.node] > > > disconnecting from > > [[nj2.client][AlhMRNg9QEqjOoZwHQKCoQ][nj2][inet[/10.126.159.169:9300]]{data=false, > > datacenter=nj, master=false}], channel closed event > > [2015-02-25 04:03:31,731][DEBUG][discovery.zen.publish] [ny1.node] > > > received cluster state version 30469 > > [2015-02-25 04:03:31,732][DEBUG][cluster.service] [ny1.node] > > > processing [zen-disco-receive(from master > > [[nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj, > > master=true}])]: execute > > [2015-02-25 04:03:31,732][DEBUG][cluster.service] [ny1.node] > > > cluster state updated, version [30469], source [zen-disco-receive(from > > master > > [[nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj, > > master=true}])] > > [2015-02-25 04:03:31,732][INFO][cluster.service] [ny1.node] > > > removed > > {[nj2.client][AlhMRNg9QEqjOoZwHQKCoQ][nj2][inet[/10.126.159.169:9300]]{data=false, > > datacenter=nj, master=false},}, reason: zen-disco-receive(from master > > [[nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj, > > master=true}]) > > [2015-02-25 04:03:31,732][DEBUG][cluster.service] [ny1.node] set > > > local cluster state to version 30469 > > [2015-02-25 04:03:31,733][DEBUG][cluster.service] [ny1.node] > > > processing [zen-disco-receive(from master > > [[nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj, > > master=true}])]: done applying updated cluster\_state (version: 30469) > > [2015-02-25 04:03:31,766][DEBUG][discovery.zen.publish] [ny1.node] > > > received cluster state version 30470 > > [2015-02-25 04:03:31,766][DEBUG][cluster.service] [ny1.node] > > > processing [zen-disco-receive(from master > > [[nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj, > > master=true}])]: execute > > [2015-02-25 04:03:31,766][DEBUG][cluster.service] [ny1.node] > > > cluster state updated, version [30470], source [zen-disco-receive(from > > master > > [[nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj, > > master=true}])] > > [2015-02-25 04:03:31,766][DEBUG][cluster.service] [ny1.node] set > > > local cluster state to version 30470 > > [2015-02-25 04:03:31,768][DEBUG][cluster.service] [ny1.node] > > > processing [zen-disco-receive(from master > > [[nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj, > > master=true}])]: done applying updated cluster\_state (version: 30470) > > [2015-02-25 04:03:31,919][INFO][discovery.zen] [ny1.node] > > > master\_left > > [[nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj, > > master=true}], reason [shut\_down] > > [2015-02-25 04:03:31,919][DEBUG][cluster.service] [ny1.node] > > > processing [zen-disco-master\_failed > > ([nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj, > > master=true})]: execute > > [2015-02-25 04:03:31,920][WARN][discovery.zen] [ny1.node] > > > master left (reason = shut\_down), current nodes: > > {[bny2.client][IO2ZNdrCSjm8M8QIswyHBQ][bny2][inet[/10.126.48.212:9300]]{data=false, > > datacenter=ny, > > master=false},[bny2.node][n1QDDYYHQYuCS7Pu\_XTD9Q][bny2][inet[/10.126.48.212:9301]]{datacenter=ny, > > master=true},[bnj1.client][6hkLZw88S5auAdoq8qoyHA][bnj1][inet[/10.126.158.81:9300]]{data=false, > > datacenter=nj, > > master=false},[ny1.node][PoLRwDtNTw2gEgV9Uz3cuA][ny1][inet[/10.126.23.55:9301]]{datacenter=ny, > > master=true},[ny1.client][v2aQp7RTQhGzXtNIrxYvbw][ny1][inet[/10.126.23.55:9300]]{data=false, > > datacenter=ny, > > master=false},[bnj1.node][YJNi4enwRE2gjolqUo4Y4Q][bnj1][inet[/10.126.158.81:9301]]{datacenter=nj, > > master=true},[bny1.client][2pqteug-S3ymIOFj567amw][bny1][inet[/10.126.48.74:9300]]{data=false, > > datacenter=ny, > > master=false},[nj1.client][DptRUOYgQnCWc829Q7gUOQ][nj1][inet[/10.126.151.178:9300]]{data=false, > > datacenter=nj, > > master=false},[ny2.node][kEESMSb0R4qMFm9vKV7Zdw][ny2][inet[/10.126.32.202:9301]]{datacenter=ny, > > master=true},[bny1.node][eSngHNhvRrSvSZ1up2H3ug][bny1][inet[/10.126.48.74:9301]]{datacenter=ny, > > master=true},[bnj2.node][v7Kjjhe3Q7mZDTUfhKafXw][bnj2][inet[/10.126.157.83:9301]]{datacenter=nj, > > master=true},[ny2.client][TNXABhLRTcqrAhCPoi6ROw][ny2][inet[/10.126.32.202:9300]]{data=false, > > datacenter=ny, > > master=false},[nj1.node][aTvj8H3bQhuWoNfXikr78Q][nj1][inet[/10.126.151.178:9301]]{datacenter=nj, > > master=true},[bnj2.client][HfoXDgx4Tm-UN2lH920iNQ][bnj2][inet[/10.126.157.83:9300]]{data=false, > > datacenter=nj, master=false},} > > [2015-02-25 04:03:31,920][DEBUG][discovery.zen.fd] [ny1.node] > > > [master] stopping fault detection against master > > [[nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj, > > master=true}], reason [master left (reason = shut\_down)] > > [2015-02-25 04:03:31,920][DEBUG][cluster.service] [ny1.node] > > > cluster state updated, version [30470], source [zen-disco-master\_failed > > ([nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj, > > master=true})] > > [2015-02-25 04:03:31,920][INFO][cluster.service] [ny1.node] > > > removed > > {[nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj, > > master=true},}, reason: zen-disco-master\_failed > > ([nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj, > > master=true}) > > [2015-02-25 04:03:31,920][DEBUG][cluster.service] [ny1.node] set > > > local cluster state to version 30470 > > [2015-02-25 04:03:31,926][DEBUG][transport.netty] [ny1.node] > > > disconnecting from > > [[nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj, > > master=true}] due to explicit disconnect call > > [2015-02-25 04:03:31,927][DEBUG][cluster.service] [ny1.node] > > > processing [zen-disco-master\_failed > > ([nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj, > > master=true})]: done applying updated cluster\_state (version: 30470) > > [2015-02-25 04:03:34,941][DEBUG][cluster.service] [ny1.node] > > > processing [zen-disco-receive(join from > > node[[bnj1.node][YJNi4enwRE2gjolqUo4Y4Q][bnj1][inet[/10.126.158.81:9301]]{datacenter=nj, > > master=true}])]: execute > > [2015-02-25 04:03:34,941][DEBUG][cluster.service] [ny1.node] > > > failing [zen-disco-receive(join from > > node[[bnj1.node][YJNi4enwRE2gjolqUo4Y4Q][bnj1][inet[/10.126.158.81:9301]]{datacenter=nj, > > master=true}])]: local node is no longer master > > [2015-02-25 04:03:34,950][DEBUG][discovery.zen] [ny1.node] > > > filtered ping responses: (filter\_client[true], filter\_data[false]) > > ``` > --> ping_response{node > > ``` > > > [[bnj2.node][v7Kjjhe3Q7mZDTUfhKafXw][bnj2][inet[/10.126.157.83:9301]]{datacenter=nj, > > master=true}], id[112], master [null], hasJoinedOnce [true], > > cluster\_name[bgov\_es]} > > ``` > --> ping_response{node > > ``` > > > [[nj1.node][aTvj8H3bQhuWoNfXikr78Q][nj1][inet[/10.126.151.178:9301]]{datacenter=nj, > > master=true}], id[95], master [null], hasJoinedOnce [true], > > cluster\_name[bgov\_es]} > > ``` > --> ping_response{node > > ``` > > > [[bny2.node][n1QDDYYHQYuCS7Pu\_XTD9Q][bny2][inet[/10.126.48.212:9301]]{datacenter=ny, > > master=true}], id[106], master [null], hasJoinedOnce [true], > > cluster\_name[bgov\_es]} > > ``` > --> ping_response{node > > ``` > > > [[bny1.node][eSngHNhvRrSvSZ1up2H3ug][bny1][inet[/10.126.48.74:9301]]{datacenter=ny, > > master=true}], id[82], master [null], hasJoinedOnce [true], > > cluster\_name[bgov\_es]} > > ``` > --> ping_response{node > > ``` > > > [[bnj1.node][YJNi4enwRE2gjolqUo4Y4Q][bnj1][inet[/10.126.158.81:9301]]{datacenter=nj, > > master=true}], id[89], master [null], hasJoinedOnce [true], > > cluster\_name[bgov\_es]} > > ``` > --> ping_response{node > > ``` > > > [[ny2.node][kEESMSb0R4qMFm9vKV7Zdw][ny2][inet[/10.126.32.202:9301]]{datacenter=ny, > > master=true}], id[76], master [null], hasJoinedOnce [true], > > cluster\_name[bgov\_es]} > > [2015-02-25 04:03:34,951][DEBUG][cluster.service] [ny1.node] > > > processing [zen-disco-receive(join from > > node[[nj1.node][aTvj8H3bQhuWoNfXikr78Q][nj1][inet[/10.126.151.178:9301]]{datacenter=nj, > > master=true}])]: execute > > [2015-02-25 04:03:34,951][DEBUG][cluster.service] [ny1.node] > > > failing [zen-disco-receive(join from > > node[[nj1.node][aTvj8H3bQhuWoNfXikr78Q][nj1][inet[/10.126.151.178:9301]]{datacenter=nj, > > master=true}])]: local node is no longer master > > [2015-02-25 04:03:34,952][DEBUG][cluster.service] [ny1.node] > > > processing [zen-disco-join (elected\_as\_master)]: execute > > [2015-02-25 04:03:35,594][DEBUG][cluster.service] [ny1.node] > > > cluster state updated, version [30471], source [zen-disco-join > > (elected\_as\_master)] > > [2015-02-25 04:03:35,594][INFO][cluster.service] [ny1.node] > > > new\_master > > [ny1.node][PoLRwDtNTw2gEgV9Uz3cuA][ny1][inet[/10.126.23.55:9301]]{datacenter=ny, > > master=true}, reason: zen-disco-join (elected\_as\_master) > > [2015-02-25 04:03:35,594][DEBUG][cluster.service] [ny1.node] > > > publishing cluster state version 30471 > > [2015-02-25 04:03:35,770][DEBUG][cluster.service] [ny1.node] set > > > local cluster state to version 30471 > > [2015-02-25 04:03:35,774][DEBUG][river.cluster] [ny1.node] > > > processing [reroute\_rivers\_node\_changed]: execute > > [2015-02-25 04:03:35,776][DEBUG][river.cluster] [ny1.node] > > > processing [reroute\_rivers\_node\_changed]: no change in cluster\_state > > [2015-02-25 04:03:35,790][DEBUG][cluster.action.shard] [ny1.node] > > > received shard started for [bgov\_exclusives\_autc\_20140619][0], > > node[YJNi4enwRE2gjolqUo4Y4Q], [R], s[INITIALIZING], indexUUID > > [-kIhiMleQgaZW03bZuj-jA], reason [after recovery (replica) from node > > [[bnj2.node][v7Kjjh > > e3Q7mZDTUfhKafXw][bnj2][inet[/10.126.157.83:9301]]{datacenter=nj, > > > master=true}]] > > [2015-02-25 04:03:35,794][DEBUG][cluster.service] [ny1.node] > > > processing [zen-disco-join (elected\_as\_master)]: done applying updated > > cluster\_state (version: 30471) On Monday, March 2, 2015 at 12:16:46 PM UTC-5, Ankush Jhalani wrote: > We have a 8 node cluster, with 1-master and 3-replica for each shard > distributed across 2 datacenters (so 2 copies of each shard/per > datacenter). When we shut-down a node for maintenance, ES takes few seconds > to re-elect new master. During this time however ES requests on _other > nodes_ fail with error - > > > {"error":"ClusterBlockException[blocked by: [SERVICE\_UNAVAILABLE/2/no > > master];]","status":503} -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/9bea576c-b26c-4c6b-affb-7fa006438d1b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9bea576c-b26c-4c6b-affb-7fa006438d1b%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Ankush\_Jhalani](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ankush_jhalani/32/875_2.png) [@Ankush\_Jhalani](https://discuss.elastic.co/u/Ankush_Jhalani)\ **Post date:** [March 2, 2015, 5:26pm UTC](https://discuss.elastic.co/t/es-failure-for-few-seconds-during-master-re-elect/22472/3 "2015-03-02T17:26:14Z")
Logs on a different node than one being restarted, when request failed - > [2015-02-25 04:03:31,701][DEBUG][transport.netty] [ny1.node] > > > disconnecting from > > [[nj2.client][AlhMRNg9QEqjOoZwHQKCoQ][nj2][inet[/10.126.159.169:9300]]{data=false, > > datacenter=nj, master=false}], channel closed event > > [2015-02-25 04:03:31,731][DEBUG][discovery.zen.publish] [ny1.node] > > > received cluster state version 30469 > > [2015-02-25 04:03:31,732][DEBUG][cluster.service] [ny1.node] > > > processing [zen-disco-receive(from master > > [[nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj, > > master=true}])]: execute > > [2015-02-25 04:03:31,732][DEBUG][cluster.service] [ny1.node] > > > cluster state updated, version [30469], source [zen-disco-receive(from > > master > > [[nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj, > > master=true}])] > > [2015-02-25 04:03:31,732][INFO][cluster.service] [ny1.node] > > > removed > > {[nj2.client][AlhMRNg9QEqjOoZwHQKCoQ][nj2][inet[/10.126.159.169:9300]]{data=false, > > datacenter=nj, master=false},}, reason: zen-disco-receive(from master > > [[nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj, > > master=true}]) > > [2015-02-25 04:03:31,732][DEBUG][cluster.service] [ny1.node] set > > > local cluster state to version 30469 > > [2015-02-25 04:03:31,733][DEBUG][cluster.service] [ny1.node] > > > processing [zen-disco-receive(from master > > [[nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj, > > master=true}])]: done applying updated cluster\_state (version: 30469) > > [2015-02-25 04:03:31,766][DEBUG][discovery.zen.publish] [ny1.node] > > > received cluster state version 30470 > > [2015-02-25 04:03:31,766][DEBUG][cluster.service] [ny1.node] > > > processing [zen-disco-receive(from master > > [[nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj, > > master=true}])]: execute > > [2015-02-25 04:03:31,766][DEBUG][cluster.service] [ny1.node] > > > cluster state updated, version [30470], source [zen-disco-receive(from > > master > > [[nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj, > > master=true}])] > > [2015-02-25 04:03:31,766][DEBUG][cluster.service] [ny1.node] set > > > local cluster state to version 30470 > > [2015-02-25 04:03:31,768][DEBUG][cluster.service] [ny1.node] > > > processing [zen-disco-receive(from master > > [[nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj, > > master=true}])]: done applying updated cluster\_state (version: 30470) > > [2015-02-25 04:03:31,919][INFO][discovery.zen] [ny1.node] > > > master\_left > > [[nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj, > > master=true}], reason [shut\_down] > > [2015-02-25 04:03:31,919][DEBUG][cluster.service] [ny1.node] > > > processing [zen-disco-master\_failed > > ([nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj, > > master=true})]: execute > > [2015-02-25 04:03:31,920][WARN][discovery.zen] [ny1.node] > > > master left (reason = shut\_down), current nodes: > > {[bny2.client][IO2ZNdrCSjm8M8QIswyHBQ][bny2][inet[/10.126.48.212:9300]]{data=false, > > datacenter=ny, > > master=false},[bny2.node][n1QDDYYHQYuCS7Pu\_XTD9Q][bny2][inet[/10.126.48.212:9301]]{datacenter=ny, > > master=true},[bnj1.client][6hkLZw88S5auAdoq8qoyHA][bnj1][inet[/10.126.158.81:9300]]{data=false, > > datacenter=nj, > > master=false},[ny1.node][PoLRwDtNTw2gEgV9Uz3cuA][ny1][inet[/10.126.23.55:9301]]{datacenter=ny, > > master=true},[ny1.client][v2aQp7RTQhGzXtNIrxYvbw][ny1][inet[/10.126.23.55:9300]]{data=false, > > datacenter=ny, > > master=false},[bnj1.node][YJNi4enwRE2gjolqUo4Y4Q][bnj1][inet[/10.126.158.81:9301]]{datacenter=nj, > > master=true},[bny1.client][2pqteug-S3ymIOFj567amw][bny1][inet[/10.126.48.74:9300]]{data=false, > > datacenter=ny, > > master=false},[nj1.client][DptRUOYgQnCWc829Q7gUOQ][nj1][inet[/10.126.151.178:9300]]{data=false, > > datacenter=nj, > > master=false},[ny2.node][kEESMSb0R4qMFm9vKV7Zdw][ny2][inet[/10.126.32.202:9301]]{datacenter=ny, > > master=true},[bny1.node][eSngHNhvRrSvSZ1up2H3ug][bny1][inet[/10.126.48.74:9301]]{datacenter=ny, > > master=true},[bnj2.node][v7Kjjhe3Q7mZDTUfhKafXw][bnj2][inet[/10.126.157.83:9301]]{datacenter=nj, > > master=true},[ny2.client][TNXABhLRTcqrAhCPoi6ROw][ny2][inet[/10.126.32.202:9300]]{data=false, > > datacenter=ny, > > master=false},[nj1.node][aTvj8H3bQhuWoNfXikr78Q][nj1][inet[/10.126.151.178:9301]]{datacenter=nj, > > master=true},[bnj2.client][HfoXDgx4Tm-UN2lH920iNQ][bnj2][inet[/10.126.157.83:9300]]{data=false, > > datacenter=nj, master=false},} > > [2015-02-25 04:03:31,920][DEBUG][discovery.zen.fd] [ny1.node] > > > [master] stopping fault detection against master > > [[nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj, > > master=true}], reason [master left (reason = shut\_down)] > > [2015-02-25 04:03:31,920][DEBUG][cluster.service] [ny1.node] > > > cluster state updated, version [30470], source [zen-disco-master\_failed > > ([nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj, > > master=true})] > > [2015-02-25 04:03:31,920][INFO][cluster.service] [ny1.node] > > > removed > > {[nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj, > > master=true},}, reason: zen-disco-master\_failed > > ([nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj, > > master=true}) > > [2015-02-25 04:03:31,920][DEBUG][cluster.service] [ny1.node] set > > > local cluster state to version 30470 > > [2015-02-25 04:03:31,926][DEBUG][transport.netty] [ny1.node] > > > disconnecting from > > [[nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj, > > master=true}] due to explicit disconnect call > > [2015-02-25 04:03:31,927][DEBUG][cluster.service] [ny1.node] > > > processing [zen-disco-master\_failed > > ([nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj, > > master=true})]: done applying updated cluster\_state (version: 30470) > > [2015-02-25 04:03:34,941][DEBUG][cluster.service] [ny1.node] > > > processing [zen-disco-receive(join from > > node[[bnj1.node][YJNi4enwRE2gjolqUo4Y4Q][bnj1][inet[/10.126.158.81:9301]]{datacenter=nj, > > master=true}])]: execute > > [2015-02-25 04:03:34,941][DEBUG][cluster.service] [ny1.node] > > > failing [zen-disco-receive(join from > > node[[bnj1.node][YJNi4enwRE2gjolqUo4Y4Q][bnj1][inet[/10.126.158.81:9301]]{datacenter=nj, > > master=true}])]: local node is no longer master > > [2015-02-25 04:03:34,950][DEBUG][discovery.zen] [ny1.node] > > > filtered ping responses: (filter\_client[true], filter\_data[false]) > > ``` > --> ping_response{node > > ``` > > > [[bnj2.node][v7Kjjhe3Q7mZDTUfhKafXw][bnj2][inet[/10.126.157.83:9301]]{datacenter=nj, > > master=true}], id[112], master [null], hasJoinedOnce [true], > > cluster\_name[es\_cluster]} > > ``` > --> ping_response{node > > ``` > > > [[nj1.node][aTvj8H3bQhuWoNfXikr78Q][nj1][inet[/10.126.151.178:9301]]{datacenter=nj, > > master=true}], id[95], master [null], hasJoinedOnce [true], > > cluster\_name[es\_cluster]} > > ``` > --> ping_response{node > > ``` > > > [[bny2.node][n1QDDYYHQYuCS7Pu\_XTD9Q][bny2][inet[/10.126.48.212:9301]]{datacenter=ny, > > master=true}], id[106], master [null], hasJoinedOnce [true], > > cluster\_name[es\_cluster]} > > ``` > --> ping_response{node > > ``` > > > [[bny1.node][eSngHNhvRrSvSZ1up2H3ug][bny1][inet[/10.126.48.74:9301]]{datacenter=ny, > > master=true}], id[82], master [null], hasJoinedOnce [true], > > cluster\_name[es\_cluster]} > > ``` > --> ping_response{node > > ``` > > > [[bnj1.node][YJNi4enwRE2gjolqUo4Y4Q][bnj1][inet[/10.126.158.81:9301]]{datacenter=nj, > > master=true}], id[89], master [null], hasJoinedOnce [true], > > cluster\_name[es\_cluster]} > > ``` > --> ping_response{node > > ``` > > > [[ny2.node][kEESMSb0R4qMFm9vKV7Zdw][ny2][inet[/10.126.32.202:9301]]{datacenter=ny, > > master=true}], id[76], master [null], hasJoinedOnce [true], > > cluster\_name[es\_cluster]} > > [2015-02-25 04:03:34,951][DEBUG][cluster.service] [ny1.node] > > > processing [zen-disco-receive(join from > > node[[nj1.node][aTvj8H3bQhuWoNfXikr78Q][nj1][inet[/10.126.151.178:9301]]{datacenter=nj, > > master=true}])]: execute > > [2015-02-25 04:03:34,951][DEBUG][cluster.service] [ny1.node] > > > failing [zen-disco-receive(join from > > node[[nj1.node][aTvj8H3bQhuWoNfXikr78Q][nj1][inet[/10.126.151.178:9301]]{datacenter=nj, > > master=true}])]: local node is no longer master > > [2015-02-25 04:03:34,952][DEBUG][cluster.service] [ny1.node] > > > processing [zen-disco-join (elected\_as\_master)]: execute > > [2015-02-25 04:03:35,594][DEBUG][cluster.service] [ny1.node] > > > cluster state updated, version [30471], source [zen-disco-join > > (elected\_as\_master)] > > [2015-02-25 04:03:35,594][INFO][cluster.service] [ny1.node] > > > new\_master > > [ny1.node][PoLRwDtNTw2gEgV9Uz3cuA][ny1][inet[/10.126.23.55:9301]]{datacenter=ny, > > master=true}, reason: zen-disco-join (elected\_as\_master) > > [2015-02-25 04:03:35,594][DEBUG][cluster.service] [ny1.node] > > > publishing cluster state version 30471 > > [2015-02-25 04:03:35,770][DEBUG][cluster.service] [ny1.node] set > > > local cluster state to version 30471 > > [2015-02-25 04:03:35,774][DEBUG][river.cluster] [ny1.node] > > > processing [reroute\_rivers\_node\_changed]: execute > > [2015-02-25 04:03:35,776][DEBUG][river.cluster] [ny1.node] > > > processing [reroute\_rivers\_node\_changed]: no change in cluster\_state > > [2015-02-25 04:03:35,790][DEBUG][cluster.action.shard] [ny1.node] > > > received shard started for [exclusives\_autc\_20140619][0], > > node[YJNi4enwRE2gjolqUo4Y4Q], [R], s[INITIALIZING], indexUUID > > [-kIhiMleQgaZW03bZuj-jA], reason [after recovery (replica) from node > > [[bnj2.node][v7Kjjh > > e3Q7mZDTUfhKafXw][bnj2][inet[/10.126.157.83:9301]]{datacenter=nj, > > > master=true}]] > > [2015-02-25 04:03:35,794][DEBUG][cluster.service] [ny1.node] > > > processing [zen-disco-join (elected\_as\_master)]: done applying updated > > cluster\_state (version: 30471) On Monday, March 2, 2015 at 12:16:46 PM UTC-5, Ankush Jhalani wrote: > We have a 8 node cluster, with 1-master and 3-replica for each shard > distributed across 2 datacenters (so 2 copies of each shard/per > datacenter). When we shut-down a node for maintenance, ES takes few seconds > to re-elect new master. During this time however ES requests on _other > nodes_ fail with error - > > > {"error":"ClusterBlockException[blocked by: [SERVICE\_UNAVAILABLE/2/no > > master];]","status":503} -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/78a8de13-9bdb-4840-bb99-c7b363505a30%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/78a8de13-9bdb-4840-bb99-c7b363505a30%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Ankush\_Jhalani](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ankush_jhalani/32/875_2.png) [@Ankush\_Jhalani](https://discuss.elastic.co/u/Ankush_Jhalani)\ **Post date:** [March 2, 2015, 5:48pm UTC](https://discuss.elastic.co/t/es-failure-for-few-seconds-during-master-re-elect/22472/4 "2015-03-02T17:48:06Z")
We are seeing the issue with recent 1.3.7. I checked our old logs with v1.2.2 and master re-election was much faster.. took few milliseconds... On Monday, March 2, 2015 at 12:16:46 PM UTC-5, Ankush Jhalani wrote: > We have a 8 node cluster, with 1-master and 3-replica for each shard > distributed across 2 datacenters (so 2 copies of each shard/per > datacenter). When we shut-down a node for maintenance, ES takes few seconds > to re-elect new master. During this time however ES requests on _other > nodes_ fail with error - > > > {"error":"ClusterBlockException[blocked by: [SERVICE\_UNAVAILABLE/2/no > > master];]","status":503} -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/e9b3d24b-3595-45fa-a4ff-b804c37edffc%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e9b3d24b-3595-45fa-a4ff-b804c37edffc%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:29am UTC](https://discuss.elastic.co/t/es-failure-for-few-seconds-during-master-re-elect/22472/5 "2017-07-06T00:29:04Z")