Remove master nodes from elasticsearch cluster

[mukularora89 (Mukul)]

Hi,

We have a requirement where we need to upgrade OS and for that Elasticsearch will be fresh installed on the nodes. I have a question related to master nodes removal.

We have 3 master nodes as part of the cluster. What approach can I take for removing master nodes from cluster for OS migration and then adding them back after OS migration is done.

Approach 1 - Remove one master node at a time for OS migration
Questions

1. Can I remove one master node, following this blog and let the elasticsearch cluster run with 2 master nodes and add the removed master nodes once OS migration is done?
2. Will this lead to split brain condition with 2 master nodes?
3. Can elasticsearch cluster will work in mixed OS mode? i.e. few master node in CentOS7 and few in RHEL9

Approach 2 - Add more master nodes to cluster with new OS and then start removing master nodes with old OS

Questions

1. Do I need to add more master nodes (with required OS) into cluster and then do the removal process one by one of master node (with old OS)?
2. If yes, what is the initial number of master nodes I have to add before start removal process considering we have 3 initial master nodes.
3. Also how these new master nodes will be added to the voting configuration?
4. Do I also need to add master nodes list "discovery.seed_hosts" for data nodes for this approach?

Please let me know your thoughts on the above approaches and which will work better.

[DavidTurner]

@mukularora89

Can I remove one master node,

Yes.

@mukularora89

Will this lead to split brain condition

No.

@mukularora89

Can elasticsearch cluster will work in mixed OS mode?

Probably yes although we don't test it.

@mukularora89

Do I need to add more master nodes

Either approach works.

@mukularora89

If yes, what is the initial number of master nodes I have to add

Up to you. You could do it one-by-one or in larger steps.

@mukularora89

Also how these new master nodes will be added to the voting configuration?

It's automatic.

@mukularora89

Do I also need to add master nodes list "discovery.seed_hosts"

Yes.

[mukularora89 (Mukul)]

Thanks David. So what I am getting from the replies that Approach 1 will work in my use case. Also with approach 1 do I need to add master node to voting configuration exclusion list, if I am removing one master at a time in a 3 master node cluster

[DavidTurner]

@mukularora89

do I need to add master node to voting configuration exclusion list, if I am removing one master at a time

Probably not, as long as you don't remove multiple master nodes too quickly.