# Replace ES master node
**URL:** https://discuss.elastic.co/t/replace-es-master-node/222987
**Category:** Elasticsearch
**Created:** [March 10, 2020, 5:21pm UTC](https://discuss.elastic.co/t/replace-es-master-node/222987 "2020-03-10T17:21:21Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@rihad](https://discuss.elastic.co/u/rihad)
#### Post date: [March 10, 2020, 5:21pm UTC](https://discuss.elastic.co/t/replace-es-master-node/222987/1 "2020-03-10T17:21:21Z")
Hi, we use a 3-node master-eligible ES6 cluster, and we need to replace one of the nodes. I'm trying to follow the official doc here: [https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-discovery-adding-removing-nodes.html](https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-discovery-adding-removing-nodes.html) and I have a few questions:
1. Is it better to bring the existing node down and then add the new one, or the other way around?
2. Is it required to let the application use the new node as soon as possible or it can get away with using its current list of nodes which does not yet have the new one?
3. Is it possible to run versions 6.8.3 & 6.8.6 in the same cluster?
Thanks!
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [March 10, 2020, 7:08pm UTC](https://discuss.elastic.co/t/replace-es-master-node/222987/2 "2020-03-10T19:08:21Z")
> [@rihad](#):
>
> 1. Is it better to bring the existing node down and then add the new one, or the other way around?
Yes, if you have an odd number of nodes, because otherwise you have to increase `discovery.zen.minimum_master_nodes` to 3 before starting the 4th master-eligible node, and then reduce it back to 2 again afterwards.
In 7.x this setting is ignored and the order doesn't matter.
> [@rihad](#):
>
> 1. Is it required to let the application use the new node as soon as possible or it can get away with using its current list of nodes which does not yet have the new one?
I don't think it matters, assuming you only have 3 nodes. If you have more nodes then your application should not be using the master-eligible ones at all.
> [@rihad](#):
>
> 1. Is it possible to run versions 6.8.3 & 6.8.6 in the same cluster?
Yes, but only while you are upgrading everything to 6.8.6. You shouldn't run a mixed cluster for an extended period of time.
---
### Author:  [@rihad](https://discuss.elastic.co/u/rihad)
#### Post date: [March 11, 2020, 5:57am UTC](https://discuss.elastic.co/t/replace-es-master-node/222987/3 "2020-03-11T05:57:03Z")
Yes, but only while you are upgrading everything to 6.8.6. You shouldn't run a mixed cluster for an extended period of time.
I think I read somewhere that minor version changes are compatible enough to be able to run fine alongside each other? Such as 6.8.3 & 6.8.6. The problem is that the other machines run older FreeBSD versions where updates aren't supported, so it will take some time to update them all, maybe a few weeks or even months.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [March 11, 2020, 6:20am UTC](https://discuss.elastic.co/t/replace-es-master-node/222987/4 "2020-03-11T06:20:39Z")
Even minor version may have different Lucerne versions and an index created on a newer node might not be possible to replicate to older nodes which can cause problems. This is the main reason as far as I know all nodes should have the same version once migration has completed.
---
### Author:  [@rihad](https://discuss.elastic.co/u/rihad)
#### Post date: [March 21, 2020, 7:31am UTC](https://discuss.elastic.co/t/replace-es-master-node/222987/5 "2020-03-21T07:31:23Z")
Ok, so if we have 3 master-eligible nodes A, B, C, and decide to add another node D (which is going to replace C), and ensure that
> [@DavidTurner](#):
>
> discovery.zen.minimum\_master\_nodes
is set to 3 before adding the server, will the client application still work with the old list A,B,C for the duration of the switch?
Then, if we replace the list of servers the client application uses to A,B,D and reload it, will the app still work? And then we would be free to just shut C down?
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [March 21, 2020, 8:46am UTC](https://discuss.elastic.co/t/replace-es-master-node/222987/6 "2020-03-21T08:46:27Z")
Yes, that sounds like it'll work.
---
### Author:  [@rihad](https://discuss.elastic.co/u/rihad)
#### Post date: [March 21, 2020, 9:08am UTC](https://discuss.elastic.co/t/replace-es-master-node/222987/7 "2020-03-21T09:08:14Z")
Thanks. There's another weird problem. With 3 ME nodes A,B,C, and minimum\_master\_nodes=2 if any of the nodes goes down, the app fails as it would with a single node, with these errors:
> [2020-03-20T23:55:07,781][DEBUG][o.e.a.s.TransportSearchAction] [[sun.example.com](http://sun.example.com)] All shards failed for phase: [query]
> org.elasticsearch.index.query.QueryShardException: failed to create query: {
Would you know what the reason could be?
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [March 21, 2020, 1:11pm UTC](https://discuss.elastic.co/t/replace-es-master-node/222987/8 "2020-03-21T13:11:49Z")
It's hard to say from this tiny fragment of a single `DEBUG` log message. Maybe this was a search that was ongoing on the node that went down, and you need to retry on a different node.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 18, 2020, 1:11pm UTC](https://discuss.elastic.co/t/replace-es-master-node/222987/9 "2020-04-18T13:11:49Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.