Working with ES 5.4
How can I know what is the maximum limit for ES master node to shutdown?
I have a 3 nodes cluster, 1 elected as master.
When I shut it down, a new master is elected.
In the meantime, I get
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master];
How can I control and estimate the maximum time until I won't get this ClusterBlockException?
My configuration is:
discovery.zen.commit_timeout: 2s
discovery.zen.publish_timeout: 2s
discovery.zen.fd.ping_timeout: 1s
transport.tcp.connect_timeout: 1s
I have a case where it took 5 seconds to find elect a master and I have a timeout of 3 in my code.
[2017-12-25T11:22:20,340][INFO ][o.e.c.s.ClusterService ] [node-1] added {{node-2}{-oIenO1hQuyzGhl2p1Im8w}{0Pik8qyjQ5OLhXWpVnJWEQ}{172.16.65.117}{172.16.65.117:9300},}, reason: zen-disco-receive(from master [master {node-3}{P_mE77gsRUOAWADrVdwM-Q}{Ol3tznBkRR2Vi58kkz1KFw}{172.16.67.71}{172.16.67.71:9300} committed version [13]])
[2017-12-25T11:25:19,691][INFO ][o.e.d.z.ZenDiscovery ] [node-1] master_left [{node-3}{P_mE77gsRUOAWADrVdwM-Q}{Ol3tznBkRR2Vi58kkz1KFw}{172.16.67.71}{172.16.67.71:9300}], reason [shut_down]
[2017-12-25T11:25:19,691][WARN ][o.e.d.z.ZenDiscovery ] [node-1] master left (reason = shut_down), current nodes: nodes:
{node-1}{x1zCNZe1QYelGTGzveOjKQ}{uk1g0qIiTZ-RxXN8StPb6w}{172.16.65.114}{172.16.65.114:9300}, local
{node-2}{-oIenO1hQuyzGhl2p1Im8w}{0Pik8qyjQ5OLhXWpVnJWEQ}{172.16.65.117}{172.16.65.117:9300}
{node-3}{P_mE77gsRUOAWADrVdwM-Q}{Ol3tznBkRR2Vi58kkz1KFw}{172.16.67.71}{172.16.67.71:9300}, master
[2017-12-25T11:25:20,580][WARN ][o.e.c.NodeConnectionsService] [node-1] failed to connect to node {node-3}{P_mE77gsRUOAWADrVdwM-Q}{Ol3tznBkRR2Vi58kkz1KFw}{172.16.67.71}{172.16.67.71:9300} (tried [1] times)
[2017-12-25T11:25:25,717][INFO ][o.e.c.s.ClusterService ] [node-1] detected_master {node-2}{-oIenO1hQuyzGhl2p1Im8w}{0Pik8qyjQ5OLhXWpVnJWEQ}{172.16.65.117}{172.16.65.117:9300}, reason: zen-disco-receive(from master [master {node-2}{-oIenO1hQuyzGhl2p1Im8w}{0Pik8qyjQ5OLhXWpVnJWEQ}{172.16.65.117}{172.16.65.117:9300} committed version [16]])
[2017-12-25T11:25:25,744][INFO ][o.e.c.s.ClusterService ] [node-1] removed {{node-3}{P_mE77gsRUOAWADrVdwM-Q}{Ol3tznBkRR2Vi58kkz1KFw}{172.16.67.71}{172.16.67.71:9300},}, reason: zen-disco-receive(from master [master {node-2}{-oIenO1hQuyzGhl2p1Im8w}{0Pik8qyjQ5OLhXWpVnJWEQ}{172.16.65.117}{172.16.65.117:9300} committed version [17]])