ES 0.20.5 adding a new node to a running cluster (unicast mode)

Elastic Stack Elasticsearch
1

hi,

I am trying to figure out how to add new nodes to a running cluster, I am
configured with unicast ZEN disco , currently 2 nodes are running on the
cluster

discovery.zen.ping.unicast.hosts: ["10.10.69.10", "10.10.69.11"]

Can I use the REST API to add the node and force discovery ? If not what is
the best way to avoid cluster recovery ?

I was thinking on the following flow:

  • Add the new IP of the new node to the above array on the first node ,
    then restart ES on this node , wait until cluster is GREEN
  • Add the nw IP of the new node to the above array on the second node,
    then restart ES on this node , wait until cluster is GREEN
  • Finally add the 3rd node with all nodes in the array , start it

Does it make sense ?

Thanks
Kobi

--
The above terms reflect a potential business arrangement, are provided solely
as a basis for further discussion, and are not intended to be and do not
constitute a legally binding obligation. No legally binding obligations will
be created, implied, or inferred until an agreement in final form is executed
in writing by all parties involved.

This email and any attachments hereto may be confidential or privileged.
If you received this communication by mistake, please don't forward it to
anyone else, please erase all copies and attachments, and please let me
know that it has gone to the wrong person. Thanks.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

OK to whom it may be in interest , I have switched back to multicast

Kobi

On Tuesday, June 18, 2013 8:29:21 PM UTC+3, Kobi Biton wrote:

hi,

I am trying to figure out how to add new nodes to a running cluster, I am
configured with unicast ZEN disco , currently 2 nodes are running on the
cluster

discovery.zen.ping.unicast.hosts: ["10.10.69.10", "10.10.69.11"]

Can I use the REST API to add the node and force discovery ? If not what
is the best way to avoid cluster recovery ?

I was thinking on the following flow:

  • Add the new IP of the new node to the above array on the first node ,
    then restart ES on this node , wait until cluster is GREEN
  • Add the nw IP of the new node to the above array on the second node,
    then restart ES on this node , wait until cluster is GREEN
  • Finally add the 3rd node with all nodes in the array , start it

Does it make sense ?

Thanks
Kobi

The above terms reflect a potential business arrangement, are provided solely
as a basis for further discussion, and are not intended to be and do not
constitute a legally binding obligation. No legally binding obligations will
be created, implied, or inferred until an agreement in final form is executed
in writing by all parties involved.

This email and any attachments hereto may be confidential or privileged.
If you received this communication by mistake, please don't forward it
to anyone else, please erase all copies and attachments, and please let
me know that it has gone to the wrong person. Thanks.

--
The above terms reflect a potential business arrangement, are provided solely
as a basis for further discussion, and are not intended to be and do not
constitute a legally binding obligation. No legally binding obligations will
be created, implied, or inferred until an agreement in final form is executed
in writing by all parties involved.

This email and any attachments hereto may be confidential or privileged.
If you received this communication by mistake, please don't forward it to
anyone else, please erase all copies and attachments, and please let me
know that it has gone to the wrong person. Thanks.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Could you please explain why you decided to go to multicast ?? did the
approach you outlined didnt work for you ?

-katta

On Thursday, 20 June 2013 14:17:12 UTC+5:30, Kobi Biton wrote:

OK to whom it may be in interest , I have switched back to multicast

Kobi

On Tuesday, June 18, 2013 8:29:21 PM UTC+3, Kobi Biton wrote:

hi,

I am trying to figure out how to add new nodes to a running cluster, I
am configured with unicast ZEN disco , currently 2 nodes are running on the
cluster

discovery.zen.ping.unicast.hosts: ["10.10.69.10", "10.10.69.11"]

Can I use the REST API to add the node and force discovery ? If not what
is the best way to avoid cluster recovery ?

I was thinking on the following flow:

  • Add the new IP of the new node to the above array on the first node ,
    then restart ES on this node , wait until cluster is GREEN
  • Add the nw IP of the new node to the above array on the second node,
    then restart ES on this node , wait until cluster is GREEN
  • Finally add the 3rd node with all nodes in the array , start it

Does it make sense ?

Thanks
Kobi

The above terms reflect a potential business arrangement, are provided solely
as a basis for further discussion, and are not intended to be and do not
constitute a legally binding obligation. No legally binding obligations will
be created, implied, or inferred until an agreement in final form is executed
in writing by all parties involved.

This email and any attachments hereto may be confidential or privileged.
If you received this communication by mistake, please don't forward it
to anyone else, please erase all copies and attachments, and please let
me know that it has gone to the wrong person. Thanks.

The above terms reflect a potential business arrangement, are provided solely
as a basis for further discussion, and are not intended to be and do not
constitute a legally binding obligation. No legally binding obligations will
be created, implied, or inferred until an agreement in final form is executed
in writing by all parties involved.

This email and any attachments hereto may be confidential or privileged.
If you received this communication by mistake, please don't forward it
to anyone else, please erase all copies and attachments, and please let
me know that it has gone to the wrong person. Thanks.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

yes of course in one work: unicast seems like an overhead ... , think
of a scenario that you need to add more nodes to the cluster (pretty
common one for us) you need to change the yml config on each node and
restart the cluster Node by Node making sure that all other nodes
"sees" the new node .... very problematic if you have a running
cluster with multiple indices and multiple shards , Maybe I am wrong
here and there is an option to use a REST call which adds the new
member and forces a ZEN discovery, that said Multicast in ES is only
used for node discovery and node master election after that is done,
the transport module kick in and nodes will communicate with each
other based on their IP and port (usually 9300) , so from my
perspective if multicast is not a limitation on your network go for
it!

Hope this bring more insight to your setup.

Kobi

On Thu, Jun 20, 2013 at 2:55 PM, Srivatsa Katta vatsa.katta@gmail.com wrote:

Could you please explain why you decided to go to multicast ?? did the
approach you outlined didnt work for you ?

-katta

On Thursday, 20 June 2013 14:17:12 UTC+5:30, Kobi Biton wrote:

OK to whom it may be in interest , I have switched back to multicast

Kobi

On Tuesday, June 18, 2013 8:29:21 PM UTC+3, Kobi Biton wrote:

hi,

I am trying to figure out how to add new nodes to a running cluster, I
am configured with unicast ZEN disco , currently 2 nodes are running on the
cluster

discovery.zen.ping.unicast.hosts: ["10.10.69.10", "10.10.69.11"]

Can I use the REST API to add the node and force discovery ? If not what
is the best way to avoid cluster recovery ?

I was thinking on the following flow:

  • Add the new IP of the new node to the above array on the first node ,
    then restart ES on this node , wait until cluster is GREEN
  • Add the nw IP of the new node to the above array on the second node,
    then restart ES on this node , wait until cluster is GREEN
  • Finally add the 3rd node with all nodes in the array , start it

Does it make sense ?

Thanks
Kobi

The above terms reflect a potential business arrangement, are provided
solely as a basis for further discussion, and are not intended to be and do
not constitute a legally binding obligation. No legally binding obligations
will be created, implied, or inferred until an agreement in final form is
executed in writing by all parties involved.

This email and any attachments hereto may be confidential or privileged.
If you received this communication by mistake, please don't forward it to
anyone else, please erase all copies and attachments, and please let me know
that it has gone to the wrong person. Thanks.

The above terms reflect a potential business arrangement, are provided
solely as a basis for further discussion, and are not intended to be and do
not constitute a legally binding obligation. No legally binding obligations
will be created, implied, or inferred until an agreement in final form is
executed in writing by all parties involved.

This email and any attachments hereto may be confidential or privileged.
If you received this communication by mistake, please don't forward it to
anyone else, please erase all copies and attachments, and please let me know
that it has gone to the wrong person. Thanks.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/GHx3OrwCORg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
The above terms reflect a potential business arrangement, are provided solely
as a basis for further discussion, and are not intended to be and do not
constitute a legally binding obligation. No legally binding obligations will
be created, implied, or inferred until an agreement in final form is executed
in writing by all parties involved.

This email and any attachments hereto may be confidential or privileged.
If you received this communication by mistake, please don't forward it to
anyone else, please erase all copies and attachments, and please let me
know that it has gone to the wrong person. Thanks.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

Kobi,
I have 2 es nodes, am trying to establish unicast discovery since my env
doesnt support multicast. I added the hosts with
discovery.zen.ping.unicast.hosts and set
discovery.zen.ping.multicast.enabled: false. Nodes are still not being
discovered by each other. I am verifying it with Marvel plugin, it shows me
Nodes: 1, not 2 as I expected. What am I missing? Thanks a lot for any help.

On Tuesday, 18 June 2013 13:29:21 UTC-4, Kobi Biton wrote:

hi,

I am trying to figure out how to add new nodes to a running cluster, I am
configured with unicast ZEN disco , currently 2 nodes are running on the
cluster

discovery.zen.ping.unicast.hosts: ["10.10.69.10", "10.10.69.11"]

Can I use the REST API to add the node and force discovery ? If not what
is the best way to avoid cluster recovery ?

I was thinking on the following flow:

  • Add the new IP of the new node to the above array on the first node ,
    then restart ES on this node , wait until cluster is GREEN
  • Add the nw IP of the new node to the above array on the second node,
    then restart ES on this node , wait until cluster is GREEN
  • Finally add the 3rd node with all nodes in the array , start it

Does it make sense ?

Thanks
Kobi

The above terms reflect a potential business arrangement, are provided solely
as a basis for further discussion, and are not intended to be and do not
constitute a legally binding obligation. No legally binding obligations will
be created, implied, or inferred until an agreement in final form is executed
in writing by all parties involved.

This email and any attachments hereto may be confidential or privileged.
If you received this communication by mistake, please don't forward it
to anyone else, please erase all copies and attachments, and please let
me know that it has gone to the wrong person. Thanks.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0a596fdf-e229-46dd-8986-582201ddb080%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

6

yes of course in one work: unicast seems like an overhead ... , think
of a scenario that you need to add more nodes to the cluster (pretty
common one for us) you need to change the yml config on each node and
restart the cluster Node by Node making sure that all other nodes
"sees" the new node ....

I'm pretty sure this is incorrect actually. I suspect if you had added some existing nodes' IPs to the discovery list on the new node, all the nodes would have formed a cluster. Information about what nodes are in the cluster is stored canonically on the master and replicated around - once a node is known about, its existence should be communicated to the other nodes.

Cheers,
Dan

Dan Fairs | dan.fairs@gmail.com | @danfairs | secondsync.com

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/E997488E-7B62-4966-9074-68F55A7345EA%40gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

7

Hey,

The unicast discovery is just a different way of doing discovery and if
configured correctly it shouldn't introduce any over head. You should think
of the list of hosts in the unicast list as a "seed list", it doesn't need
to be complete. When a node starts it will ping these addresses to register
it self and discover all other nodes currently in the cluster. After this
initial discovery bootstrapping, the nodes gossip with each other and do
not use this list.

Typically you'd put a short list there, like 3, of nodes that you expect to
always be part of your cluster. Note that it's OK to temporarily take one
of those nodes off line, as the rest of the list will be used as fall back.

Cheers,
Boaz

On Monday, February 3, 2014 9:44:20 PM UTC+1, Dan Fairs wrote:

yes of course in one work: unicast seems like an overhead ... , think
of a scenario that you need to add more nodes to the cluster (pretty
common one for us) you need to change the yml config on each node and
restart the cluster Node by Node making sure that all other nodes
"sees" the new node ....

I'm pretty sure this is incorrect actually. I suspect if you had added
some existing nodes' IPs to the discovery list on the new node, all the
nodes would have formed a cluster. Information about what nodes are in the
cluster is stored canonically on the master and replicated around - once a
node is known about, its existence should be communicated to the other
nodes.

Cheers,
Dan

Dan Fairs | dan....@gmail.com <javascript:> | @danfairs | secondsync.com

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6539b1f3-7ebd-4133-84ad-c27e68b36166%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.