Remove master nodes from elasticsearch cluster

Elastic Stack Elasticsearch
1

Hi,

We have a requirement where we need to upgrade OS and for that Elasticsearch will be fresh installed on the nodes. I have a question related to master nodes removal.

We have 3 master nodes as part of the cluster. What approach can I take for removing master nodes from cluster for OS migration and then adding them back after OS migration is done.

Approach 1 - Remove one master node at a time for OS migration
Questions

  1. Can I remove one master node, following this blog and let the elasticsearch cluster run with 2 master nodes and add the removed master nodes once OS migration is done?
  2. Will this lead to split brain condition with 2 master nodes?
  3. Can elasticsearch cluster will work in mixed OS mode? i.e. few master node in CentOS7 and few in RHEL9

Approach 2 - Add more master nodes to cluster with new OS and then start removing master nodes with old OS

Questions

  1. Do I need to add more master nodes (with required OS) into cluster and then do the removal process one by one of master node (with old OS)?
  2. If yes, what is the initial number of master nodes I have to add before start removal process considering we have 3 initial master nodes.
  3. Also how these new master nodes will be added to the voting configuration?
  4. Do I also need to add master nodes list "discovery.seed_hosts" for data nodes for this approach?

Please let me know your thoughts on the above approaches and which will work better.

2

Yes.

No.

Probably yes although we don't test it.

Either approach works.

Up to you. You could do it one-by-one or in larger steps.

It's automatic.

Yes.

3

Thanks David. So what I am getting from the replies that Approach 1 will work in my use case. Also with approach 1 do I need to add master node to voting configuration exclusion list, if I am removing one master at a time in a 3 master node cluster

4

Probably not, as long as you don't remove multiple master nodes too quickly.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.