How does master node works?

Elastic Stack Elasticsearch
1

I am planning to setup a large elasticsearch cluster (eg. 100 machines)
Each machine having a few ES jvm instances running. Out of all these instances, 3 will be dedicated master instances. Others will be dedicated data instances

If one of the master instance is dead. How would the cluster respond? Is there any performance degradation?

How do i properly recover the dead master? Is there a guide in doing this? (Eg. clean up data folder)

2

Hi,

If one of the master instance is dead. How would the cluster respond?

It the master eligible node that died was the elected master, an election would occur to elect a new master among the remaining master eligible nodes.
(There is only 1 master (elected master) and many eligible masters)

Is there any performance degradation?

It depend what you mean I guess, in an ES cluster the elected master node is in the critical path of some actions that are critical for the cluster to function correctly but without going into too much details that you should get direct from the documentation... It's fine when the elected master dies, that's a supported event if you will. It can happen at 3 am while you sleep. The election is fast and all is well in a manner of seconds. Some things just keep chugging along even during elections. So it complex but the short answer is that is a supported failure mode. (Loosing the elected master abruptly.)

How do i properly recover the dead master?

It can stay dead and never come back, in the sense that you could replace it with an entirely new eligible master ES node that has never seen daylight(data) before. It will join and bring the number of eligible masters back to 3, it will sync everything it needs from the elected master and be ready to take over like all the other master eligible nodes in case its ever needed. If it's the same node that died that comes back, like a reboot or process crash it also works without your intervention, it rejoins basically.

Is there a guide in doing this? (Eg. clean up data folder)

https://www.elastic.co/guide/en/elasticsearch/reference/current/discovery-settings.html

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.