Force change cluster master

Elastic Stack Elasticsearch
1

Hi,

Today I've been trying to increase the storage of my elasticsearch nodes. The nodes are hosted on AWS EC2 each with an attached EBS of 10 GB. I was trying to increase the EBS size to 20GB for each node and it all went fine until I went and restarted the cluster master.

It took about 30 seconds for the cluster to elect a new master and during that time all requests failed and all the other node gave me the 503 error when I tried to check their status.

I am wondering if there is a way to change the cluster master to a specific node instantly without having to wait for the nodes to elect a new master.

For example, lets say my cluster has three nodes:

Node1 (Cluster Master)
Node2
Node3

What I would like to do is change the size of the drive for lets say Node2, then once that node joins the cluster and all shards get reallocated, I force elect it as the cluster master, then I can safely change the configuration for the other two nodes.

Is this possible in ES? If not, how can I go about reducing the time it takes for the nodes to elect a new master?

1 Like
2

You cannot do this. You have to restart the existing master.

4

But this means my cluster will be down until a new master gets elected.

So if this is not possible, then how can I minimize the down time?

5

Is there a recommended solution for the above, or did you figure out a way to resolve this? Each time I want to update my ES version, I can do so on all the nodes, but as soon as I restart my master node, I get a 503 as you were until it elects a new master? Did you found a workaround to this?

6

Unfortunately no, I just moved on from this as it doesn't seem possible for the time being. Hopefully elastic can do some new API in upcoming versions.

7

Thanks for your reply @Zaid_Amir . Hopefully they'll provide a solution to this at some point.

8

During Rolling Restart of ES if we rebuild the master node , then there will be impacts to our System until we scale up the last node and new master is elected.(Impacts will be there for 30 seconds)
Only solution is to force elect other node as cluster master and then do rebuild on this specific node.
But it seems not possible in ElasticSearch?
Please let us know whether the solution is provided?

9

Could you force a change of master without restarting a node by temporarily blocking the connection to that node (simulating a network partition).

How long would it take for the rest of the cluster to elect a new master? after that time you could remove the blockage and get the node to join the cluster again.

I know it is an old topic, but it is quite interesting.

10

You'd be better off just restarting the node, it'd have the same impact with less hassle.

it should have been closed (ie archived)! :slight_smile:

11

Why do you feel you need to do this? If this is related to this thread I suspect changing the current master node is unlikely to help as it is possible that it is the propagation of the cluster state update that is taking time.

12

I had the same question yesterday, to see if I could force a change of master to a different node. Mostly because in our cluster, elastic shares resources with lots of different services, and the node that was master was also very busy (with a zeppelin), and so changing the leadership to another node would have been nice, and restarting wasn't an option (messing with the communication wasn't either).
I'll reply to the other thread.

2 Likes