# Add / Remove nodes in cluster, good practice question

**URL:** https://discuss.elastic.co/t/add-remove-nodes-in-cluster-good-practice-question/18825
**Category:** Elasticsearch
**Created:** [July 23, 2014, 2:21pm UTC](https://discuss.elastic.co/t/add-remove-nodes-in-cluster-good-practice-question/18825 "2014-07-23T14:21:31Z")
**Posts on this page:** 5
**Page:** 1

<div class="post-metadata">

### Author: ![Pierre\_Vincent\_Ledou](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierre_vincent_ledou/32/1368_2.png) [@Pierre\_Vincent\_Ledou](https://discuss.elastic.co/u/Pierre_Vincent_Ledou)
#### Post date: [July 23, 2014, 2:21pm UTC](https://discuss.elastic.co/t/add-remove-nodes-in-cluster-good-practice-question/18825/1 "2014-07-23T14:21:31Z")

</div>

Hi,

We have a little cluster of 2 nodes, hosting 4 indexes of about 1.5M  
documents each, replicated on both nodes.

Those 2 nodes are on VPS that are stored on the same physical host. As it  
represents a single point of failure, we have decided to start a new VPS on  
a different host.

What is the correct procedure to add the new node to the cluster, get the  
indexes replicated in it, and then removed one of the older node?

We don't use mutlicast, so I imagine that I can add the node the the  
unicast list in the config file, but how I can be sure that it will not  
fail all my cluster when I will restart elasticsearch?

Those nodes are online in production so it's a bit touchy for us to take  
any risk with it.

Cheers,

Pv

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1681d317-f0a1-4bff-9256-eabf1de2a3cc%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1681d317-f0a1-4bff-9256-eabf1de2a3cc%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [July 23, 2014, 10:58pm UTC](https://discuss.elastic.co/t/add-remove-nodes-in-cluster-good-practice-question/18825/2 "2014-07-23T22:58:17Z")

</div>

If this is production you really want an odd number of nodes to reduce  
potential split brain issues.

However in your case, just add the new node to the cluster, let it  
replicate across, then shutdown the node you no longer want. Any impact  
will be minimal.

Regards,  
Mark Walkom

Infrastructure Engineer  
Campaign Monitor  
email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com)  
web: [www.campaignmonitor.com](http://www.campaignmonitor.com)

On 24 July 2014 00:21, Pierre-Vincent Ledoux [pvledoux@gmail.com](mailto:pvledoux@gmail.com) wrote:

> Hi,
> 
> We have a little cluster of 2 nodes, hosting 4 indexes of about 1.5M  
> documents each, replicated on both nodes.
> 
> Those 2 nodes are on VPS that are stored on the same physical host. As it  
> represents a single point of failure, we have decided to start a new VPS on  
> a different host.
> 
> What is the correct procedure to add the new node to the cluster, get the  
> indexes replicated in it, and then removed one of the older node?
> 
> We don't use mutlicast, so I imagine that I can add the node the the  
> unicast list in the config file, but how I can be sure that it will not  
> fail all my cluster when I will restart elasticsearch?
> 
> Those nodes are online in production so it's a bit touchy for us to take  
> any risk with it.
> 
> Cheers,
> 
> Pv
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/1681d317-f0a1-4bff-9256-eabf1de2a3cc%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1681d317-f0a1-4bff-9256-eabf1de2a3cc%40googlegroups.com)  
> [https://groups.google.com/d/msgid/elasticsearch/1681d317-f0a1-4bff-9256-eabf1de2a3cc%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/1681d317-f0a1-4bff-9256-eabf1de2a3cc%40googlegroups.com?utm_medium=email&utm_source=footer)  
> .  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624ak4b-%2BUOw-1KD0s8Pyvjdv%2BrEmbYXUeXsittqqPfKv%3Dg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624ak4b-%2BUOw-1KD0s8Pyvjdv%2BrEmbYXUeXsittqqPfKv%3Dg%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

### Author: ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)
#### Post date: [July 24, 2014, 2:06am UTC](https://discuss.elastic.co/t/add-remove-nodes-in-cluster-good-practice-question/18825/3 "2014-07-24T02:06:59Z")

</div>

1. Fix the unicast discovery list to contain the two nodes you'll end up  
with. Make sure the cluster name matches.
2. Start the new node.
3. Watch it join the cluster. If it doesn't then check step 1 and repeat 2  
until it works.
4. Use the allocation filtering API to ban all shards from the node you  
will be removing.
5. Wait until there are no shards on that node. You can check using the  
\_cat/shards API.
6. Shut down the node.

You might want to think about getting three nodes and configuring minimum  
master nodes but I'm sure you've heard that before.

Nik  
On Jul 23, 2014 10:21 AM, "Pierre-Vincent Ledoux" [pvledoux@gmail.com](mailto:pvledoux@gmail.com)  
wrote:

> Hi,
> 
> We have a little cluster of 2 nodes, hosting 4 indexes of about 1.5M  
> documents each, replicated on both nodes.
> 
> Those 2 nodes are on VPS that are stored on the same physical host. As it  
> represents a single point of failure, we have decided to start a new VPS on  
> a different host.
> 
> What is the correct procedure to add the new node to the cluster, get the  
> indexes replicated in it, and then removed one of the older node?
> 
> We don't use mutlicast, so I imagine that I can add the node the the  
> unicast list in the config file, but how I can be sure that it will not  
> fail all my cluster when I will restart elasticsearch?
> 
> Those nodes are online in production so it's a bit touchy for us to take  
> any risk with it.
> 
> Cheers,
> 
> Pv
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/1681d317-f0a1-4bff-9256-eabf1de2a3cc%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1681d317-f0a1-4bff-9256-eabf1de2a3cc%40googlegroups.com)  
> [https://groups.google.com/d/msgid/elasticsearch/1681d317-f0a1-4bff-9256-eabf1de2a3cc%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/1681d317-f0a1-4bff-9256-eabf1de2a3cc%40googlegroups.com?utm_medium=email&utm_source=footer)  
> .  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd0L2oP0KzeVjZ\_5%2BWoeAC%3D4%2Bb\_FVkhdVhn\_UtC6vmLMxg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd0L2oP0KzeVjZ_5%2BWoeAC%3D4%2Bb_FVkhdVhn_UtC6vmLMxg%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

### Author: ![Pierre\_Vincent\_Ledou](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierre_vincent_ledou/32/1368_2.png) [@Pierre\_Vincent\_Ledou](https://discuss.elastic.co/u/Pierre_Vincent_Ledou)
#### Post date: [July 24, 2014, 10:12am UTC](https://discuss.elastic.co/t/add-remove-nodes-in-cluster-good-practice-question/18825/4 "2014-07-24T10:12:06Z")

</div>

Hi Mark,

thanks! Worked like a charm!

Best regards,

Pv

On 24 Jul 2014, at 00:58, Mark Walkom [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) wrote:

> If this is production you really want an odd number of nodes to reduce potential split brain issues.
> 
> However in your case, just add the new node to the cluster, let it replicate across, then shutdown the node you no longer want. Any impact will be minimal.
> 
> Regards,  
> Mark Walkom
> 
> Infrastructure Engineer  
> Campaign Monitor  
> email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com)  
> web: [www.campaignmonitor.com](http://www.campaignmonitor.com)
> 
> On 24 July 2014 00:21, Pierre-Vincent Ledoux [pvledoux@gmail.com](mailto:pvledoux@gmail.com) wrote:  
> Hi,
> 
> We have a little cluster of 2 nodes, hosting 4 indexes of about 1.5M documents each, replicated on both nodes.
> 
> Those 2 nodes are on VPS that are stored on the same physical host. As it represents a single point of failure, we have decided to start a new VPS on a different host.
> 
> What is the correct procedure to add the new node to the cluster, get the indexes replicated in it, and then removed one of the older node?
> 
> We don't use mutlicast, so I imagine that I can add the node the the unicast list in the config file, but how I can be sure that it will not fail all my cluster when I will restart elasticsearch?
> 
> Those nodes are online in production so it's a bit touchy for us to take any risk with it.
> 
> Cheers,
> 
> Pv
> 
> --  
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1681d317-f0a1-4bff-9256-eabf1de2a3cc%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1681d317-f0a1-4bff-9256-eabf1de2a3cc%40googlegroups.com).  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> 
> --  
> You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.  
> To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/7VWJwyNUqvA/unsubscribe](https://groups.google.com/d/topic/elasticsearch/7VWJwyNUqvA/unsubscribe).  
> To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624ak4b-%2BUOw-1KD0s8Pyvjdv%2BrEmbYXUeXsittqqPfKv%3Dg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624ak4b-%2BUOw-1KD0s8Pyvjdv%2BrEmbYXUeXsittqqPfKv%3Dg%40mail.gmail.com).  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 1:13am UTC](https://discuss.elastic.co/t/add-remove-nodes-in-cluster-good-practice-question/18825/5 "2017-07-06T01:13:34Z")

</div>


