Add / Remove nodes in cluster, good practice question

Elastic Stack Elasticsearch
1

Hi,

We have a little cluster of 2 nodes, hosting 4 indexes of about 1.5M
documents each, replicated on both nodes.

Those 2 nodes are on VPS that are stored on the same physical host. As it
represents a single point of failure, we have decided to start a new VPS on
a different host.

What is the correct procedure to add the new node to the cluster, get the
indexes replicated in it, and then removed one of the older node?

We don't use mutlicast, so I imagine that I can add the node the the
unicast list in the config file, but how I can be sure that it will not
fail all my cluster when I will restart elasticsearch?

Those nodes are online in production so it's a bit touchy for us to take
any risk with it.

Cheers,

Pv

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1681d317-f0a1-4bff-9256-eabf1de2a3cc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

If this is production you really want an odd number of nodes to reduce
potential split brain issues.

However in your case, just add the new node to the cluster, let it
replicate across, then shutdown the node you no longer want. Any impact
will be minimal.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 24 July 2014 00:21, Pierre-Vincent Ledoux pvledoux@gmail.com wrote:

Hi,

We have a little cluster of 2 nodes, hosting 4 indexes of about 1.5M
documents each, replicated on both nodes.

Those 2 nodes are on VPS that are stored on the same physical host. As it
represents a single point of failure, we have decided to start a new VPS on
a different host.

What is the correct procedure to add the new node to the cluster, get the
indexes replicated in it, and then removed one of the older node?

We don't use mutlicast, so I imagine that I can add the node the the
unicast list in the config file, but how I can be sure that it will not
fail all my cluster when I will restart elasticsearch?

Those nodes are online in production so it's a bit touchy for us to take
any risk with it.

Cheers,

Pv

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1681d317-f0a1-4bff-9256-eabf1de2a3cc%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/1681d317-f0a1-4bff-9256-eabf1de2a3cc%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624ak4b-%2BUOw-1KD0s8Pyvjdv%2BrEmbYXUeXsittqqPfKv%3Dg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3
  1. Fix the unicast discovery list to contain the two nodes you'll end up
    with. Make sure the cluster name matches.
  2. Start the new node.
  3. Watch it join the cluster. If it doesn't then check step 1 and repeat 2
    until it works.
  4. Use the allocation filtering API to ban all shards from the node you
    will be removing.
  5. Wait until there are no shards on that node. You can check using the
    _cat/shards API.
  6. Shut down the node.

You might want to think about getting three nodes and configuring minimum
master nodes but I'm sure you've heard that before.

Nik
On Jul 23, 2014 10:21 AM, "Pierre-Vincent Ledoux" pvledoux@gmail.com
wrote:

Hi,

We have a little cluster of 2 nodes, hosting 4 indexes of about 1.5M
documents each, replicated on both nodes.

Those 2 nodes are on VPS that are stored on the same physical host. As it
represents a single point of failure, we have decided to start a new VPS on
a different host.

What is the correct procedure to add the new node to the cluster, get the
indexes replicated in it, and then removed one of the older node?

We don't use mutlicast, so I imagine that I can add the node the the
unicast list in the config file, but how I can be sure that it will not
fail all my cluster when I will restart elasticsearch?

Those nodes are online in production so it's a bit touchy for us to take
any risk with it.

Cheers,

Pv

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1681d317-f0a1-4bff-9256-eabf1de2a3cc%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/1681d317-f0a1-4bff-9256-eabf1de2a3cc%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd0L2oP0KzeVjZ_5%2BWoeAC%3D4%2Bb_FVkhdVhn_UtC6vmLMxg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

4

Hi Mark,

thanks! Worked like a charm!

Best regards,

Pv

On 24 Jul 2014, at 00:58, Mark Walkom markw@campaignmonitor.com wrote:

If this is production you really want an odd number of nodes to reduce potential split brain issues.

However in your case, just add the new node to the cluster, let it replicate across, then shutdown the node you no longer want. Any impact will be minimal.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 24 July 2014 00:21, Pierre-Vincent Ledoux pvledoux@gmail.com wrote:
Hi,

We have a little cluster of 2 nodes, hosting 4 indexes of about 1.5M documents each, replicated on both nodes.

Those 2 nodes are on VPS that are stored on the same physical host. As it represents a single point of failure, we have decided to start a new VPS on a different host.

What is the correct procedure to add the new node to the cluster, get the indexes replicated in it, and then removed one of the older node?

We don't use mutlicast, so I imagine that I can add the node the the unicast list in the config file, but how I can be sure that it will not fail all my cluster when I will restart elasticsearch?

Those nodes are online in production so it's a bit touchy for us to take any risk with it.

Cheers,

Pv

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1681d317-f0a1-4bff-9256-eabf1de2a3cc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/7VWJwyNUqvA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624ak4b-%2BUOw-1KD0s8Pyvjdv%2BrEmbYXUeXsittqqPfKv%3Dg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.