ES failure for few seconds during master re-elect

Elastic Stack Elasticsearch
1

We have a 8 node cluster, with 1-master and 3-replica for each shard
distributed across 2 datacenters (so 2 copies of each shard/per
datacenter). When we shut-down a node for maintenance, ES takes few seconds
to re-elect new master. During this time however ES requests on other
nodes fail with error -

{"error":"ClusterBlockException[blocked by: [SERVICE_UNAVAILABLE/2/no
master];]","status":503}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/46811878-c8d5-4c31-b10a-5c66b6ad1673%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Logs on a node which failed ES requests -

[2015-02-25 04:03:31,701][DEBUG][transport.netty ] [ny1.node]

disconnecting from
[[nj2.client][AlhMRNg9QEqjOoZwHQKCoQ][nj2][inet[/10.126.159.169:9300]]{data=false,
datacenter=nj, master=false}], channel closed event

[2015-02-25 04:03:31,731][DEBUG][discovery.zen.publish ] [ny1.node]

received cluster state version 30469

[2015-02-25 04:03:31,732][DEBUG][cluster.service ] [ny1.node]

processing [zen-disco-receive(from master
[[nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj,
master=true}])]: execute

[2015-02-25 04:03:31,732][DEBUG][cluster.service ] [ny1.node]

cluster state updated, version [30469], source [zen-disco-receive(from
master
[[nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj,
master=true}])]

[2015-02-25 04:03:31,732][INFO ][cluster.service ] [ny1.node]

removed
{[nj2.client][AlhMRNg9QEqjOoZwHQKCoQ][nj2][inet[/10.126.159.169:9300]]{data=false,
datacenter=nj, master=false},}, reason: zen-disco-receive(from master
[[nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj,
master=true}])

[2015-02-25 04:03:31,732][DEBUG][cluster.service ] [ny1.node] set

local cluster state to version 30469

[2015-02-25 04:03:31,733][DEBUG][cluster.service ] [ny1.node]

processing [zen-disco-receive(from master
[[nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj,
master=true}])]: done applying updated cluster_state (version: 30469)

[2015-02-25 04:03:31,766][DEBUG][discovery.zen.publish ] [ny1.node]

received cluster state version 30470

[2015-02-25 04:03:31,766][DEBUG][cluster.service ] [ny1.node]

processing [zen-disco-receive(from master
[[nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj,
master=true}])]: execute

[2015-02-25 04:03:31,766][DEBUG][cluster.service ] [ny1.node]

cluster state updated, version [30470], source [zen-disco-receive(from
master
[[nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj,
master=true}])]

[2015-02-25 04:03:31,766][DEBUG][cluster.service ] [ny1.node] set

local cluster state to version 30470

[2015-02-25 04:03:31,768][DEBUG][cluster.service ] [ny1.node]

processing [zen-disco-receive(from master
[[nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj,
master=true}])]: done applying updated cluster_state (version: 30470)

[2015-02-25 04:03:31,919][INFO ][discovery.zen ] [ny1.node]

master_left
[[nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj,
master=true}], reason [shut_down]

[2015-02-25 04:03:31,919][DEBUG][cluster.service ] [ny1.node]

processing [zen-disco-master_failed
([nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj,
master=true})]: execute

[2015-02-25 04:03:31,920][WARN ][discovery.zen ] [ny1.node]

master left (reason = shut_down), current nodes:
{[bny2.client][IO2ZNdrCSjm8M8QIswyHBQ][bny2][inet[/10.126.48.212:9300]]{data=false,
datacenter=ny,
master=false},[bny2.node][n1QDDYYHQYuCS7Pu_XTD9Q][bny2][inet[/10.126.48.212:9301]]{datacenter=ny,
master=true},[bnj1.client][6hkLZw88S5auAdoq8qoyHA][bnj1][inet[/10.126.158.81:9300]]{data=false,
datacenter=nj,
master=false},[ny1.node][PoLRwDtNTw2gEgV9Uz3cuA][ny1][inet[/10.126.23.55:9301]]{datacenter=ny,
master=true},[ny1.client][v2aQp7RTQhGzXtNIrxYvbw][ny1][inet[/10.126.23.55:9300]]{data=false,
datacenter=ny,
master=false},[bnj1.node][YJNi4enwRE2gjolqUo4Y4Q][bnj1][inet[/10.126.158.81:9301]]{datacenter=nj,
master=true},[bny1.client][2pqteug-S3ymIOFj567amw][bny1][inet[/10.126.48.74:9300]]{data=false,
datacenter=ny,
master=false},[nj1.client][DptRUOYgQnCWc829Q7gUOQ][nj1][inet[/10.126.151.178:9300]]{data=false,
datacenter=nj,
master=false},[ny2.node][kEESMSb0R4qMFm9vKV7Zdw][ny2][inet[/10.126.32.202:9301]]{datacenter=ny,
master=true},[bny1.node][eSngHNhvRrSvSZ1up2H3ug][bny1][inet[/10.126.48.74:9301]]{datacenter=ny,
master=true},[bnj2.node][v7Kjjhe3Q7mZDTUfhKafXw][bnj2][inet[/10.126.157.83:9301]]{datacenter=nj,
master=true},[ny2.client][TNXABhLRTcqrAhCPoi6ROw][ny2][inet[/10.126.32.202:9300]]{data=false,
datacenter=ny,
master=false},[nj1.node][aTvj8H3bQhuWoNfXikr78Q][nj1][inet[/10.126.151.178:9301]]{datacenter=nj,
master=true},[bnj2.client][HfoXDgx4Tm-UN2lH920iNQ][bnj2][inet[/10.126.157.83:9300]]{data=false,
datacenter=nj, master=false},}

[2015-02-25 04:03:31,920][DEBUG][discovery.zen.fd ] [ny1.node]

[master] stopping fault detection against master
[[nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj,
master=true}], reason [master left (reason = shut_down)]

[2015-02-25 04:03:31,920][DEBUG][cluster.service ] [ny1.node]

cluster state updated, version [30470], source [zen-disco-master_failed
([nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj,
master=true})]

[2015-02-25 04:03:31,920][INFO ][cluster.service ] [ny1.node]

removed
{[nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj,
master=true},}, reason: zen-disco-master_failed
([nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj,
master=true})

[2015-02-25 04:03:31,920][DEBUG][cluster.service ] [ny1.node] set

local cluster state to version 30470

[2015-02-25 04:03:31,926][DEBUG][transport.netty ] [ny1.node]

disconnecting from
[[nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj,
master=true}] due to explicit disconnect call

[2015-02-25 04:03:31,927][DEBUG][cluster.service ] [ny1.node]

processing [zen-disco-master_failed
([nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj,
master=true})]: done applying updated cluster_state (version: 30470)

[2015-02-25 04:03:34,941][DEBUG][cluster.service ] [ny1.node]

processing [zen-disco-receive(join from
node[[bnj1.node][YJNi4enwRE2gjolqUo4Y4Q][bnj1][inet[/10.126.158.81:9301]]{datacenter=nj,
master=true}])]: execute

[2015-02-25 04:03:34,941][DEBUG][cluster.service ] [ny1.node]

failing [zen-disco-receive(join from
node[[bnj1.node][YJNi4enwRE2gjolqUo4Y4Q][bnj1][inet[/10.126.158.81:9301]]{datacenter=nj,
master=true}])]: local node is no longer master

[2015-02-25 04:03:34,950][DEBUG][discovery.zen ] [ny1.node]

filtered ping responses: (filter_client[true], filter_data[false])

    --> ping_response{node

[[bnj2.node][v7Kjjhe3Q7mZDTUfhKafXw][bnj2][inet[/10.126.157.83:9301]]{datacenter=nj,
master=true}], id[112], master [null], hasJoinedOnce [true],
cluster_name[bgov_es]}

    --> ping_response{node

[[nj1.node][aTvj8H3bQhuWoNfXikr78Q][nj1][inet[/10.126.151.178:9301]]{datacenter=nj,
master=true}], id[95], master [null], hasJoinedOnce [true],
cluster_name[bgov_es]}

    --> ping_response{node

[[bny2.node][n1QDDYYHQYuCS7Pu_XTD9Q][bny2][inet[/10.126.48.212:9301]]{datacenter=ny,
master=true}], id[106], master [null], hasJoinedOnce [true],
cluster_name[bgov_es]}

    --> ping_response{node

[[bny1.node][eSngHNhvRrSvSZ1up2H3ug][bny1][inet[/10.126.48.74:9301]]{datacenter=ny,
master=true}], id[82], master [null], hasJoinedOnce [true],
cluster_name[bgov_es]}

    --> ping_response{node

[[bnj1.node][YJNi4enwRE2gjolqUo4Y4Q][bnj1][inet[/10.126.158.81:9301]]{datacenter=nj,
master=true}], id[89], master [null], hasJoinedOnce [true],
cluster_name[bgov_es]}

    --> ping_response{node

[[ny2.node][kEESMSb0R4qMFm9vKV7Zdw][ny2][inet[/10.126.32.202:9301]]{datacenter=ny,
master=true}], id[76], master [null], hasJoinedOnce [true],
cluster_name[bgov_es]}

[2015-02-25 04:03:34,951][DEBUG][cluster.service ] [ny1.node]

processing [zen-disco-receive(join from
node[[nj1.node][aTvj8H3bQhuWoNfXikr78Q][nj1][inet[/10.126.151.178:9301]]{datacenter=nj,
master=true}])]: execute

[2015-02-25 04:03:34,951][DEBUG][cluster.service ] [ny1.node]

failing [zen-disco-receive(join from
node[[nj1.node][aTvj8H3bQhuWoNfXikr78Q][nj1][inet[/10.126.151.178:9301]]{datacenter=nj,
master=true}])]: local node is no longer master

[2015-02-25 04:03:34,952][DEBUG][cluster.service ] [ny1.node]

processing [zen-disco-join (elected_as_master)]: execute

[2015-02-25 04:03:35,594][DEBUG][cluster.service ] [ny1.node]

cluster state updated, version [30471], source [zen-disco-join
(elected_as_master)]

[2015-02-25 04:03:35,594][INFO ][cluster.service ] [ny1.node]

new_master
[ny1.node][PoLRwDtNTw2gEgV9Uz3cuA][ny1][inet[/10.126.23.55:9301]]{datacenter=ny,
master=true}, reason: zen-disco-join (elected_as_master)

[2015-02-25 04:03:35,594][DEBUG][cluster.service ] [ny1.node]

publishing cluster state version 30471

[2015-02-25 04:03:35,770][DEBUG][cluster.service ] [ny1.node] set

local cluster state to version 30471

[2015-02-25 04:03:35,774][DEBUG][river.cluster ] [ny1.node]

processing [reroute_rivers_node_changed]: execute

[2015-02-25 04:03:35,776][DEBUG][river.cluster ] [ny1.node]

processing [reroute_rivers_node_changed]: no change in cluster_state

[2015-02-25 04:03:35,790][DEBUG][cluster.action.shard ] [ny1.node]

received shard started for [bgov_exclusives_autc_20140619][0],
node[YJNi4enwRE2gjolqUo4Y4Q], [R], s[INITIALIZING], indexUUID
[-kIhiMleQgaZW03bZuj-jA], reason [after recovery (replica) from node
[[bnj2.node][v7Kjjh

e3Q7mZDTUfhKafXw][bnj2][inet[/10.126.157.83:9301]]{datacenter=nj,

master=true}]]

[2015-02-25 04:03:35,794][DEBUG][cluster.service ] [ny1.node]

processing [zen-disco-join (elected_as_master)]: done applying updated
cluster_state (version: 30471)

On Monday, March 2, 2015 at 12:16:46 PM UTC-5, Ankush Jhalani wrote:

We have a 8 node cluster, with 1-master and 3-replica for each shard
distributed across 2 datacenters (so 2 copies of each shard/per
datacenter). When we shut-down a node for maintenance, ES takes few seconds
to re-elect new master. During this time however ES requests on other
nodes fail with error -

{"error":"ClusterBlockException[blocked by: [SERVICE_UNAVAILABLE/2/no
master];]","status":503}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9bea576c-b26c-4c6b-affb-7fa006438d1b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

3

Logs on a different node than one being restarted, when request failed -

[2015-02-25 04:03:31,701][DEBUG][transport.netty ] [ny1.node]

disconnecting from
[[nj2.client][AlhMRNg9QEqjOoZwHQKCoQ][nj2][inet[/10.126.159.169:9300]]{data=false,
datacenter=nj, master=false}], channel closed event

[2015-02-25 04:03:31,731][DEBUG][discovery.zen.publish ] [ny1.node]

received cluster state version 30469

[2015-02-25 04:03:31,732][DEBUG][cluster.service ] [ny1.node]

processing [zen-disco-receive(from master
[[nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj,
master=true}])]: execute

[2015-02-25 04:03:31,732][DEBUG][cluster.service ] [ny1.node]

cluster state updated, version [30469], source [zen-disco-receive(from
master
[[nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj,
master=true}])]

[2015-02-25 04:03:31,732][INFO ][cluster.service ] [ny1.node]

removed
{[nj2.client][AlhMRNg9QEqjOoZwHQKCoQ][nj2][inet[/10.126.159.169:9300]]{data=false,
datacenter=nj, master=false},}, reason: zen-disco-receive(from master
[[nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj,
master=true}])

[2015-02-25 04:03:31,732][DEBUG][cluster.service ] [ny1.node] set

local cluster state to version 30469

[2015-02-25 04:03:31,733][DEBUG][cluster.service ] [ny1.node]

processing [zen-disco-receive(from master
[[nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj,
master=true}])]: done applying updated cluster_state (version: 30469)

[2015-02-25 04:03:31,766][DEBUG][discovery.zen.publish ] [ny1.node]

received cluster state version 30470

[2015-02-25 04:03:31,766][DEBUG][cluster.service ] [ny1.node]

processing [zen-disco-receive(from master
[[nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj,
master=true}])]: execute

[2015-02-25 04:03:31,766][DEBUG][cluster.service ] [ny1.node]

cluster state updated, version [30470], source [zen-disco-receive(from
master
[[nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj,
master=true}])]

[2015-02-25 04:03:31,766][DEBUG][cluster.service ] [ny1.node] set

local cluster state to version 30470

[2015-02-25 04:03:31,768][DEBUG][cluster.service ] [ny1.node]

processing [zen-disco-receive(from master
[[nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj,
master=true}])]: done applying updated cluster_state (version: 30470)

[2015-02-25 04:03:31,919][INFO ][discovery.zen ] [ny1.node]

master_left
[[nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj,
master=true}], reason [shut_down]

[2015-02-25 04:03:31,919][DEBUG][cluster.service ] [ny1.node]

processing [zen-disco-master_failed
([nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj,
master=true})]: execute

[2015-02-25 04:03:31,920][WARN ][discovery.zen ] [ny1.node]

master left (reason = shut_down), current nodes:
{[bny2.client][IO2ZNdrCSjm8M8QIswyHBQ][bny2][inet[/10.126.48.212:9300]]{data=false,
datacenter=ny,
master=false},[bny2.node][n1QDDYYHQYuCS7Pu_XTD9Q][bny2][inet[/10.126.48.212:9301]]{datacenter=ny,
master=true},[bnj1.client][6hkLZw88S5auAdoq8qoyHA][bnj1][inet[/10.126.158.81:9300]]{data=false,
datacenter=nj,
master=false},[ny1.node][PoLRwDtNTw2gEgV9Uz3cuA][ny1][inet[/10.126.23.55:9301]]{datacenter=ny,
master=true},[ny1.client][v2aQp7RTQhGzXtNIrxYvbw][ny1][inet[/10.126.23.55:9300]]{data=false,
datacenter=ny,
master=false},[bnj1.node][YJNi4enwRE2gjolqUo4Y4Q][bnj1][inet[/10.126.158.81:9301]]{datacenter=nj,
master=true},[bny1.client][2pqteug-S3ymIOFj567amw][bny1][inet[/10.126.48.74:9300]]{data=false,
datacenter=ny,
master=false},[nj1.client][DptRUOYgQnCWc829Q7gUOQ][nj1][inet[/10.126.151.178:9300]]{data=false,
datacenter=nj,
master=false},[ny2.node][kEESMSb0R4qMFm9vKV7Zdw][ny2][inet[/10.126.32.202:9301]]{datacenter=ny,
master=true},[bny1.node][eSngHNhvRrSvSZ1up2H3ug][bny1][inet[/10.126.48.74:9301]]{datacenter=ny,
master=true},[bnj2.node][v7Kjjhe3Q7mZDTUfhKafXw][bnj2][inet[/10.126.157.83:9301]]{datacenter=nj,
master=true},[ny2.client][TNXABhLRTcqrAhCPoi6ROw][ny2][inet[/10.126.32.202:9300]]{data=false,
datacenter=ny,
master=false},[nj1.node][aTvj8H3bQhuWoNfXikr78Q][nj1][inet[/10.126.151.178:9301]]{datacenter=nj,
master=true},[bnj2.client][HfoXDgx4Tm-UN2lH920iNQ][bnj2][inet[/10.126.157.83:9300]]{data=false,
datacenter=nj, master=false},}

[2015-02-25 04:03:31,920][DEBUG][discovery.zen.fd ] [ny1.node]

[master] stopping fault detection against master
[[nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj,
master=true}], reason [master left (reason = shut_down)]

[2015-02-25 04:03:31,920][DEBUG][cluster.service ] [ny1.node]

cluster state updated, version [30470], source [zen-disco-master_failed
([nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj,
master=true})]

[2015-02-25 04:03:31,920][INFO ][cluster.service ] [ny1.node]

removed
{[nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj,
master=true},}, reason: zen-disco-master_failed
([nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj,
master=true})

[2015-02-25 04:03:31,920][DEBUG][cluster.service ] [ny1.node] set

local cluster state to version 30470

[2015-02-25 04:03:31,926][DEBUG][transport.netty ] [ny1.node]

disconnecting from
[[nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj,
master=true}] due to explicit disconnect call

[2015-02-25 04:03:31,927][DEBUG][cluster.service ] [ny1.node]

processing [zen-disco-master_failed
([nj2.node][RGXiGOUIQYSkaGhPX1dDDg][nj2][inet[/10.126.159.169:9301]]{datacenter=nj,
master=true})]: done applying updated cluster_state (version: 30470)

[2015-02-25 04:03:34,941][DEBUG][cluster.service ] [ny1.node]

processing [zen-disco-receive(join from
node[[bnj1.node][YJNi4enwRE2gjolqUo4Y4Q][bnj1][inet[/10.126.158.81:9301]]{datacenter=nj,
master=true}])]: execute

[2015-02-25 04:03:34,941][DEBUG][cluster.service ] [ny1.node]

failing [zen-disco-receive(join from
node[[bnj1.node][YJNi4enwRE2gjolqUo4Y4Q][bnj1][inet[/10.126.158.81:9301]]{datacenter=nj,
master=true}])]: local node is no longer master

[2015-02-25 04:03:34,950][DEBUG][discovery.zen ] [ny1.node]

filtered ping responses: (filter_client[true], filter_data[false])

    --> ping_response{node

[[bnj2.node][v7Kjjhe3Q7mZDTUfhKafXw][bnj2][inet[/10.126.157.83:9301]]{datacenter=nj,
master=true}], id[112], master [null], hasJoinedOnce [true],
cluster_name[es_cluster]}

    --> ping_response{node

[[nj1.node][aTvj8H3bQhuWoNfXikr78Q][nj1][inet[/10.126.151.178:9301]]{datacenter=nj,
master=true}], id[95], master [null], hasJoinedOnce [true],
cluster_name[es_cluster]}

    --> ping_response{node

[[bny2.node][n1QDDYYHQYuCS7Pu_XTD9Q][bny2][inet[/10.126.48.212:9301]]{datacenter=ny,
master=true}], id[106], master [null], hasJoinedOnce [true],
cluster_name[es_cluster]}

    --> ping_response{node

[[bny1.node][eSngHNhvRrSvSZ1up2H3ug][bny1][inet[/10.126.48.74:9301]]{datacenter=ny,
master=true}], id[82], master [null], hasJoinedOnce [true],
cluster_name[es_cluster]}

    --> ping_response{node

[[bnj1.node][YJNi4enwRE2gjolqUo4Y4Q][bnj1][inet[/10.126.158.81:9301]]{datacenter=nj,
master=true}], id[89], master [null], hasJoinedOnce [true],
cluster_name[es_cluster]}

    --> ping_response{node

[[ny2.node][kEESMSb0R4qMFm9vKV7Zdw][ny2][inet[/10.126.32.202:9301]]{datacenter=ny,
master=true}], id[76], master [null], hasJoinedOnce [true],
cluster_name[es_cluster]}

[2015-02-25 04:03:34,951][DEBUG][cluster.service ] [ny1.node]

processing [zen-disco-receive(join from
node[[nj1.node][aTvj8H3bQhuWoNfXikr78Q][nj1][inet[/10.126.151.178:9301]]{datacenter=nj,
master=true}])]: execute

[2015-02-25 04:03:34,951][DEBUG][cluster.service ] [ny1.node]

failing [zen-disco-receive(join from
node[[nj1.node][aTvj8H3bQhuWoNfXikr78Q][nj1][inet[/10.126.151.178:9301]]{datacenter=nj,
master=true}])]: local node is no longer master

[2015-02-25 04:03:34,952][DEBUG][cluster.service ] [ny1.node]

processing [zen-disco-join (elected_as_master)]: execute

[2015-02-25 04:03:35,594][DEBUG][cluster.service ] [ny1.node]

cluster state updated, version [30471], source [zen-disco-join
(elected_as_master)]

[2015-02-25 04:03:35,594][INFO ][cluster.service ] [ny1.node]

new_master
[ny1.node][PoLRwDtNTw2gEgV9Uz3cuA][ny1][inet[/10.126.23.55:9301]]{datacenter=ny,
master=true}, reason: zen-disco-join (elected_as_master)

[2015-02-25 04:03:35,594][DEBUG][cluster.service ] [ny1.node]

publishing cluster state version 30471

[2015-02-25 04:03:35,770][DEBUG][cluster.service ] [ny1.node] set

local cluster state to version 30471

[2015-02-25 04:03:35,774][DEBUG][river.cluster ] [ny1.node]

processing [reroute_rivers_node_changed]: execute

[2015-02-25 04:03:35,776][DEBUG][river.cluster ] [ny1.node]

processing [reroute_rivers_node_changed]: no change in cluster_state

[2015-02-25 04:03:35,790][DEBUG][cluster.action.shard ] [ny1.node]

received shard started for [exclusives_autc_20140619][0],
node[YJNi4enwRE2gjolqUo4Y4Q], [R], s[INITIALIZING], indexUUID
[-kIhiMleQgaZW03bZuj-jA], reason [after recovery (replica) from node
[[bnj2.node][v7Kjjh

e3Q7mZDTUfhKafXw][bnj2][inet[/10.126.157.83:9301]]{datacenter=nj,

master=true}]]

[2015-02-25 04:03:35,794][DEBUG][cluster.service ] [ny1.node]

processing [zen-disco-join (elected_as_master)]: done applying updated
cluster_state (version: 30471)

On Monday, March 2, 2015 at 12:16:46 PM UTC-5, Ankush Jhalani wrote:

We have a 8 node cluster, with 1-master and 3-replica for each shard
distributed across 2 datacenters (so 2 copies of each shard/per
datacenter). When we shut-down a node for maintenance, ES takes few seconds
to re-elect new master. During this time however ES requests on other
nodes fail with error -

{"error":"ClusterBlockException[blocked by: [SERVICE_UNAVAILABLE/2/no
master];]","status":503}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/78a8de13-9bdb-4840-bb99-c7b363505a30%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

We are seeing the issue with recent 1.3.7. I checked our old logs with
v1.2.2 and master re-election was much faster.. took few milliseconds...

On Monday, March 2, 2015 at 12:16:46 PM UTC-5, Ankush Jhalani wrote:

We have a 8 node cluster, with 1-master and 3-replica for each shard
distributed across 2 datacenters (so 2 copies of each shard/per
datacenter). When we shut-down a node for maintenance, ES takes few seconds
to re-elect new master. During this time however ES requests on other
nodes fail with error -

{"error":"ClusterBlockException[blocked by: [SERVICE_UNAVAILABLE/2/no
master];]","status":503}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e9b3d24b-3595-45fa-a4ff-b804c37edffc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.