Clarification around unicast host discovery w/ round-robin dns

Elastic Stack Elasticsearch
1

Our environment does not support multicast, therefore we are doing manual
host discovery with something like:

discovery.zen.ping.unicast.hosts
["es1.ourdomain.com:9300","es2.ourdomain.com:9300"]

The question I would like answered is whether or not ES can use a single
address bound to multiple IP's via around-robin DNS as the host in the
configuration.

Such as:

discovery.zen.ping.unicast.hosts ["cluster.ourdomain.com:9300"]

where cluster.ourdomain.com has multiple A records for round-robin DNS
pointing to all of our ES instances.

My experience is that ES asks for the first IP, and if it is the machine's
own IP the node will fail to join the cluster and never request another IP
from DNS. Is this behavior accurate or am I taking crazy pills?

Thank you.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2ae14b57-1fbf-497c-a529-30da48b191a0%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

1 Like
2

It'll use whatever IP is handed to it by DNS, so if it's it's own then yes
it will fail.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 6 December 2013 06:46, Kevin S kevinsteger@gmail.com wrote:

Our environment does not support multicast, therefore we are doing manual
host discovery with something like:

discovery.zen.ping.unicast.hosts ["es1.ourdomain.com:9300","
es2.ourdomain.com:9300"]

The question I would like answered is whether or not ES can use a single
address bound to multiple IP's via around-robin DNS as the host in the
configuration.

Such as:

discovery.zen.ping.unicast.hosts ["cluster.ourdomain.com:9300"]

where cluster.ourdomain.com has multiple A records for round-robin DNS
pointing to all of our ES instances.

My experience is that ES asks for the first IP, and if it is the machine's
own IP the node will fail to join the cluster and never request another IP
from DNS. Is this behavior accurate or am I taking crazy pills?

Thank you.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/2ae14b57-1fbf-497c-a529-30da48b191a0%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624aap58GU%3DvQX1MqzhvvqVt%3Dy%2BUtYk3iAvBfABOo9kuDcA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Hey,

you should definately go with specyfying concrete hosts instead of using
aliases DNS names. Also, if you do not want to specify all the nodes of
your cluster in unicast.hosts, just specify a few (maybe the master nodes),
where you are always sure they are up and running. If one of the nodes in
that list can be connected to, the node will successfully join the cluster
and get all the needed information from the cluster state from then on.

--Alex

On Thu, Dec 5, 2013 at 8:46 PM, Kevin S kevinsteger@gmail.com wrote:

Our environment does not support multicast, therefore we are doing manual
host discovery with something like:

discovery.zen.ping.unicast.hosts ["es1.ourdomain.com:9300","
es2.ourdomain.com:9300"]

The question I would like answered is whether or not ES can use a single
address bound to multiple IP's via around-robin DNS as the host in the
configuration.

Such as:

discovery.zen.ping.unicast.hosts ["cluster.ourdomain.com:9300"]

where cluster.ourdomain.com has multiple A records for round-robin DNS
pointing to all of our ES instances.

My experience is that ES asks for the first IP, and if it is the machine's
own IP the node will fail to join the cluster and never request another IP
from DNS. Is this behavior accurate or am I taking crazy pills?

Thank you.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/2ae14b57-1fbf-497c-a529-30da48b191a0%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-Ew1kcshpxQUVLRcRvv51re04sN%3D6oND72697jRKBngA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

I would prefer to automate the scale-out of our cluster and use a
homogeneous elasticsearch image. (we use docker for this and it works well
for masterless distributed services)

Having a master that you know is always up does not really align with the
cluster where "anyone can be a master" design.

I'm adding this as a feature request. Simply stated: If DNS fails, try
again and get another IP in the cluster.

Thanks for everyone's clarification on this.

On Thursday, December 5, 2013 11:49:35 PM UTC-8, Alexander Reelsen wrote:

Hey,

you should definately go with specyfying concrete hosts instead of using
aliases DNS names. Also, if you do not want to specify all the nodes of
your cluster in unicast.hosts, just specify a few (maybe the master nodes),
where you are always sure they are up and running. If one of the nodes in
that list can be connected to, the node will successfully join the cluster
and get all the needed information from the cluster state from then on.

--Alex

On Thu, Dec 5, 2013 at 8:46 PM, Kevin S <kevin...@gmail.com <javascript:>>wrote:

Our environment does not support multicast, therefore we are doing manual
host discovery with something like:

discovery.zen.ping.unicast.hosts ["es1.ourdomain.com:9300","
es2.ourdomain.com:9300"]

The question I would like answered is whether or not ES can use a single
address bound to multiple IP's via around-robin DNS as the host in the
configuration.

Such as:

discovery.zen.ping.unicast.hosts ["cluster.ourdomain.com:9300"]

where cluster.ourdomain.com has multiple A records for round-robin DNS
pointing to all of our ES instances.

My experience is that ES asks for the first IP, and if it is the
machine's own IP the node will fail to join the cluster and never request
another IP from DNS. Is this behavior accurate or am I taking crazy pills?

Thank you.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/2ae14b57-1fbf-497c-a529-30da48b191a0%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8f340615-1a04-432c-8f9f-231564801cbf%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

1 Like
5

How do you find docker? Have you tried virtualisation instead, and if so
how was the comparison?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 11 December 2013 05:28, Kevin S kevinsteger@gmail.com wrote:

I would prefer to automate the scale-out of our cluster and use a
homogeneous elasticsearch image. (we use docker for this and it works well
for masterless distributed services)

Having a master that you know is always up does not really align with the
cluster where "anyone can be a master" design.

I'm adding this as a feature request. Simply stated: If DNS fails, try
again and get another IP in the cluster.

Thanks for everyone's clarification on this.

On Thursday, December 5, 2013 11:49:35 PM UTC-8, Alexander Reelsen wrote:

Hey,

you should definately go with specyfying concrete hosts instead of using
aliases DNS names. Also, if you do not want to specify all the nodes of
your cluster in unicast.hosts, just specify a few (maybe the master nodes),
where you are always sure they are up and running. If one of the nodes in
that list can be connected to, the node will successfully join the cluster
and get all the needed information from the cluster state from then on.

--Alex

On Thu, Dec 5, 2013 at 8:46 PM, Kevin S kevin...@gmail.com wrote:

Our environment does not support multicast, therefore we are doing
manual host discovery with something like:

discovery.zen.ping.unicast.hosts ["es1.ourdomain.com:9300","es2
.ourdomain.com:9300"]

The question I would like answered is whether or not ES can use a single
address bound to multiple IP's via around-robin DNS as the host in the
configuration.

Such as:

discovery.zen.ping.unicast.hosts ["cluster.ourdomain.com:9300"]

where cluster.ourdomain.com has multiple A records for round-robin DNS
pointing to all of our ES instances.

My experience is that ES asks for the first IP, and if it is the
machine's own IP the node will fail to join the cluster and never request
another IP from DNS. Is this behavior accurate or am I taking crazy pills?

Thank you.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/2ae14b57-1fbf-497c-a529-30da48b191a0%
40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8f340615-1a04-432c-8f9f-231564801cbf%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624Y%2BBBA7hG8ETAvVMp-Xxde_YuKxn3mTVrmPBggSGV1Dqw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

6

Might sound harsh, but that is exactly what multicast in zero-config
networking was invented for...

Jörg

On Tue, Dec 10, 2013 at 7:28 PM, Kevin S kevinsteger@gmail.com wrote:

I'm adding this as a feature request. Simply stated: If DNS fails, try
again and get another IP in the cluster.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHkS8jPjSJZ8yuY7-ZM60n6MShhNbZ3%2BVy4KSdvtzLF1w%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

7

As the original post states, our environment does not support multicast.

I'm also going to wager that lack of multicast will be more and more common
as infrastructure moves to cloud instances.

Kevin

On Tuesday, December 10, 2013 4:31:47 PM UTC-8, Jörg Prante wrote:

Might sound harsh, but that is exactly what multicast in zero-config
networking was invented for...

Jörg

On Tue, Dec 10, 2013 at 7:28 PM, Kevin S <kevin...@gmail.com <javascript:>

wrote:

I'm adding this as a feature request. Simply stated: If DNS fails, try
again and get another IP in the cluster.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e59d6212-d58e-41aa-8391-e6e8799f40db%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

1 Like
8

There are products that advertise they can offer virtual private networks
to integrate cloud networks, including UDP multicast:

http://www.cohesiveft.com/products/vns3/

Amazon EC2 can be tricked to return multicast-like results with pseudo L2
broadcast:

But how this can be instrumented for Elasticsearch, I can't say.

In the end it is just a matter how much effort one wants to spend to bring
up the necessary L2 layer services in the cloud.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG4Nk_SQ7Gn%2BJXW6EH532gwwynTsU03W2duxTwKDecdLA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

9

I will look at the L2 multicast products but this seems like heavy
engineering in place of a consistent, simple, and expected behavior.

Elasticsearch already exhibits this design principle with ports. e.g. if
Port 9200 is busy, go to 9201. My expectation based on this design is that
it would be present in discovery services. e.g. if the ping fails, try a
second.

Many of the community clients offer this behavior as well, with a "fail
fast" attempt on a node then move on to the next.

This design would also work very well with those who manage with Route 53
round-robin and built in health checks.

My experience with the ES community is to say yes to simple enhancements
that make the product more robust and easier to adopt in various
environments.

Kevin

On Tuesday, December 10, 2013 6:11:56 PM UTC-8, Jörg Prante wrote:

There are products that advertise they can offer virtual private networks
to integrate cloud networks, including UDP multicast:

Lost Redirection

Amazon EC2 can be tricked to return multicast-like results with pseudo L2
broadcast:

IP Multicast on ec2 | PPT

But how this can be instrumented for Elasticsearch, I can't say.

In the end it is just a matter how much effort one wants to spend to bring
up the necessary L2 layer services in the cloud.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2c1cc82e-d8c8-4313-98a7-dba237336a97%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

10

Heavy engineering? Not really. It is just some piece of software for cloud
networking. Something like Rackspace did with "cloud networks":
http://blog.rackspace.co.uk/rackspace-news/cloud-networks-the-next-chapter-in-the-open-cloud/
and I'm sure there is more activity going on anywhere else, this is
just a
random example. The cloud industry knows it must provide products so
customers can deploy standard network technologies, and protocol support
for broadcast/multicast is a fundamental one.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFhFzSxD7-tMZOUzKDe3-dL1wwKMMn60bz5WULpwoavHg%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

11

Multicast is evil, especially in hybrid or heterogenous environments. In
cloud environments you can't generally control the underlying network
topology and, in fact, in a successful cloud infrastructure (as the
consumer) you shouldn't have to care. Kevin's request is spot-on. Asking
a node of an ES cluster to look for some other node than itself from a dns
request isn't unreasonable -- it is a bit odd but doesn't seem a bad
trade-off in order to enable run-anywhere rather than run anywhere you can
roll your own multicast network. What Rackspace is providing is
essentially Software Define Networks, Jorg (and hella lot of code)-- good
stuff, and can support multicast, but at this point if you're developing
applications that require multi-case or block storage instead of relying on
an underlying piece of infrastructure that my have those dependencies
you're probably doing it wrong -- or you're actually developing something
that low level in which case "good on ya" and I hope your stuff makes it to
stable so we can all use it.

Taking ES the way of multicast would be a huge mistake -- enabling it to
smartly join a self-mediating/eventually consistent set of homogenous
instances with minimal overhead seems reasonable.

Having a flag in ES that simply indicates the node is part of a set of
nodes including itself, might be a good start. It begs the question of
what happens when any of the nodes in the DNS chain fail to respond, of
course. :wink:

On Wednesday, December 11, 2013 12:15:44 PM UTC-8, Jörg Prante wrote:

Heavy engineering? Not really. It is just some piece of software for cloud
networking. Something like Rackspace did with "cloud networks":
http://blog.rackspace.co.uk/rackspace-news/cloud-networks-the-next-chapter-in-the-open-cloud/http://www.google.com/url?q=http%3A%2F%2Fblog.rackspace.co.uk%2Frackspace-news%2Fcloud-networks-the-next-chapter-in-the-open-cloud%2F&sa=D&sntz=1&usg=AFQjCNFwu1r1bvLo4qh3w0wCIEG58sps4g and I'm sure there is more activity going on anywhere else, this is just a
random example. The cloud industry knows it must provide products so
customers can deploy standard network technologies, and protocol support
for broadcast/multicast is a fundamental one.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/32bedff4-c42b-4541-b082-29542d3fc5c9%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

1 Like
12

On Tuesday, 10 December 2013 19:28:25 UTC+1, Kevin S wrote:

I would prefer to automate the scale-out of our cluster and use a
homogeneous elasticsearch image. (we use docker for this and it works well
for masterless distributed services)

I'm with Kevin on this one. Host discovery via unicast round-robin DNS
would be useful in the numerous scenarios where multicast is less
desirable, and I frankly see no reason for Elasticsearch not to support
this. It is simple, convenient, predictable, widely supported, very
manageable and well-behaved functionality.

One could always get the full array of A records and consider it a list of
nodes.

I'm not much of a java developer, but I expect this could also be done via
a plugin through the existing API?

--
Roy Olsen
Lead DBA, Xait

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4b1a7142-6e7e-46c7-897c-00bcafc89e7e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

1 Like
13

Same here, we planned out a masterless infrastructure thinking we could use RRDNS, like any other tool, and the fact that we can't is a drawback.