Can ES correctly re-balance shards when all master nodes down?

Elastic Stack Elasticsearch
1

Our cluster has 3 dedicated master nodes, 3 dedicated client nodes and several data nodes.

I know ES query still works even all master nodes down. But what will happen when some of data nodes down? Can ES still re-balance the shards from those down data nodes?

2

No, it won't. If all master eligible nodes are unavailable then the other nodes will stop serving requests as they have nothing maintaining the cluster.

If only data nodes go down and there is adequate master nodes available, then yes ES will rebalance.

3

HI,
ES node goes down means it shows unresponsiveness or noes is shut down.

4

Thanks your reply.
Per our testing, the query DID work after we shutdown all dedicated master nodes. Our query endpoints is on dedicated client nodes.

As for re-balancing, in my understanding, re-balancing need master nodes involvement, so if all mater nodes down, ES could not do shard re-balance when any data node down.

But the problem is, if all master nodes down, the head plugin as well as _cat API won't work, I am not able to verify whether the shards are re-balanced or NOT.

Do you know how to verify?

5

Then you must have other nodes that can become masters.

6

NO. I am 100% sure that all my master eligible nodes were down.

7

Can you post the output from _cat/nodes?

8

the problem is, when all master nodes down, the _cat API does not work, although the query still works.
BTW: I sent all request to client nodes.