Load balancing ElasticSearch with virtual ip?

Hi,

I am setting up a new environment with ElasticSearch, and the out of the
box zen discovery makes it real easy to load balance.

But if I want to set up my environment like the following.

Three servers running on the same network, works perfectly and they find
each other.

es1 10.2.88.21
es2 10.2.88.22
es3 10.2.88.23

But then, from my web application I want to access the cluster via ip
10.2.88.24, which is a virtual ip so that if one node goes down, it does
not matter since I always access 10.2.88.24 and it contains all three of
them.

Is my intentions clear? How would i configure ElasticSearch for this
purpose?

//Tommy

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Just use a DNS CNAME that points to all 3 hosts.
ie elastic-cluster.domain = es1.domain, es2.domain, es3.domain.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 23 July 2013 21:32, Tommy Albinsson tommy.albinsson@gmail.com wrote:

Hi,

I am setting up a new environment with ElasticSearch, and the out of the
box zen discovery makes it real easy to load balance.

But if I want to set up my environment like the following.

Three servers running on the same network, works perfectly and they find
each other.

es1 10.2.88.21
es2 10.2.88.22
es3 10.2.88.23

But then, from my web application I want to access the cluster via ip
10.2.88.24, which is a virtual ip so that if one node goes down, it does
not matter since I always access 10.2.88.24 and it contains all three of
them.

Is my intentions clear? How would i configure ElasticSearch for this
purpose?

//Tommy

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hmms, is it that simple?

If one of the nodes goes down for example es1.domain it does not matter
then?

That would be just awesome if it was the case.

//Tommy

Den tisdagen den 23:e juli 2013 kl. 13:48:58 UTC+2 skrev Mark Walkom:

Just use a DNS CNAME that points to all 3 hosts.
ie elastic-cluster.domain = es1.domain, es2.domain, es3.domain.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com <javascript:>
web: www.campaignmonitor.com

On 23 July 2013 21:32, Tommy Albinsson <tommy.a...@gmail.com <javascript:>

wrote:

Hi,

I am setting up a new environment with ElasticSearch, and the out of the
box zen discovery makes it real easy to load balance.

But if I want to set up my environment like the following.

Three servers running on the same network, works perfectly and they find
each other.

es1 10.2.88.21
es2 10.2.88.22
es3 10.2.88.23

But then, from my web application I want to access the cluster via ip
10.2.88.24, which is a virtual ip so that if one node goes down, it does
not matter since I always access 10.2.88.24 and it contains all three of
them.

Is my intentions clear? How would i configure ElasticSearch for this
purpose?

//Tommy

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

So it does not matter if ElasticSearch is on port 90200?

If I add elastic-cluster.domain as you describe and access it for example
via elast-cluster.domain:90200, that would be correct woldn't it?

//Tommy

Den tisdagen den 23:e juli 2013 kl. 14:27:30 UTC+2 skrev Tommy Albinsson:

Hmms, is it that simple?

If one of the nodes goes down for example es1.domain it does not matter
then?

That would be just awesome if it was the case.

//Tommy

Den tisdagen den 23:e juli 2013 kl. 13:48:58 UTC+2 skrev Mark Walkom:

Just use a DNS CNAME that points to all 3 hosts.
ie elastic-cluster.domain = es1.domain, es2.domain, es3.domain.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 23 July 2013 21:32, Tommy Albinsson tommy.a...@gmail.com wrote:

Hi,

I am setting up a new environment with ElasticSearch, and the out of the
box zen discovery makes it real easy to load balance.

But if I want to set up my environment like the following.

Three servers running on the same network, works perfectly and they find
each other.

es1 10.2.88.21
es2 10.2.88.22
es3 10.2.88.23

But then, from my web application I want to access the cluster via ip
10.2.88.24, which is a virtual ip so that if one node goes down, it does
not matter since I always access 10.2.88.24 and it contains all three of
them.

Is my intentions clear? How would i configure ElasticSearch for this
purpose?

//Tommy

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

If you want a virtual IP, use ucarp. I use it with success, in front of
nginx, JBoss, httpd, whatever.

http://www.pureftpd.org/project/ucarp

Jörg

On Tue, Jul 23, 2013 at 2:31 PM, Tommy Albinsson
tommy.albinsson@gmail.comwrote:

So it does not matter if ElasticSearch is on port 90200?

If I add elastic-cluster.domain as you describe and access it for example
via elast-cluster.domain:90200, that would be correct woldn't it?

//Tommy

Den tisdagen den 23:e juli 2013 kl. 14:27:30 UTC+2 skrev Tommy Albinsson:

Hmms, is it that simple?

If one of the nodes goes down for example es1.domain it does not matter
then?

That would be just awesome if it was the case.

//Tommy

Den tisdagen den 23:e juli 2013 kl. 13:48:58 UTC+2 skrev Mark Walkom:

Just use a DNS CNAME that points to all 3 hosts.
ie elastic-cluster.domain = es1.domain, es2.domain, es3.domain.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 23 July 2013 21:32, Tommy Albinsson tommy.a...@gmail.com wrote:

Hi,

I am setting up a new environment with ElasticSearch, and the out of
the box zen discovery makes it real easy to load balance.

But if I want to set up my environment like the following.

Three servers running on the same network, works perfectly and they
find each other.

es1 10.2.88.21
es2 10.2.88.22
es3 10.2.88.23

But then, from my web application I want to access the cluster via ip
10.2.88.24, which is a virtual ip so that if one node goes down, it does
not matter since I always access 10.2.88.24 and it contains all three of
them.

Is my intentions clear? How would i configure ElasticSearch for this
purpose?

//Tommy

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.**com.
For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

It does not have to be a virtual ip, I just want to use the same url to
access my cluster and regarding what node goes down it should not matter.

//Tommy

Den tisdagen den 23:e juli 2013 kl. 14:41:40 UTC+2 skrev Jörg Prante:

If you want a virtual IP, use ucarp. I use it with success, in front of
nginx, JBoss, httpd, whatever.

http://www.pureftpd.org/project/ucarp

Jörg

On Tue, Jul 23, 2013 at 2:31 PM, Tommy Albinsson <tommy.a...@gmail.com<javascript:>

wrote:

So it does not matter if ElasticSearch is on port 90200?

If I add elastic-cluster.domain as you describe and access it for example
via elast-cluster.domain:90200, that would be correct woldn't it?

//Tommy

Den tisdagen den 23:e juli 2013 kl. 14:27:30 UTC+2 skrev Tommy Albinsson:

Hmms, is it that simple?

If one of the nodes goes down for example es1.domain it does not matter
then?

That would be just awesome if it was the case.

//Tommy

Den tisdagen den 23:e juli 2013 kl. 13:48:58 UTC+2 skrev Mark Walkom:

Just use a DNS CNAME that points to all 3 hosts.
ie elastic-cluster.domain = es1.domain, es2.domain, es3.domain.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 23 July 2013 21:32, Tommy Albinsson tommy.a...@gmail.com wrote:

Hi,

I am setting up a new environment with ElasticSearch, and the out of
the box zen discovery makes it real easy to load balance.

But if I want to set up my environment like the following.

Three servers running on the same network, works perfectly and they
find each other.

es1 10.2.88.21
es2 10.2.88.22
es3 10.2.88.23

But then, from my web application I want to access the cluster via ip
10.2.88.24, which is a virtual ip so that if one node goes down, it does
not matter since I always access 10.2.88.24 and it contains all three of
them.

Is my intentions clear? How would i configure ElasticSearch for this
purpose?

//Tommy

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.**com.
For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

I am on a windows environment also :slight_smile:

//Tommy

Den tisdagen den 23:e juli 2013 kl. 14:58:19 UTC+2 skrev Tommy Albinsson:

It does not have to be a virtual ip, I just want to use the same url to
access my cluster and regarding what node goes down it should not matter.

//Tommy

Den tisdagen den 23:e juli 2013 kl. 14:41:40 UTC+2 skrev Jörg Prante:

If you want a virtual IP, use ucarp. I use it with success, in front of
nginx, JBoss, httpd, whatever.

http://www.pureftpd.org/project/ucarp

Jörg

On Tue, Jul 23, 2013 at 2:31 PM, Tommy Albinsson tommy.a...@gmail.comwrote:

So it does not matter if ElasticSearch is on port 90200?

If I add elastic-cluster.domain as you describe and access it for
example via elast-cluster.domain:90200, that would be correct woldn't it?

//Tommy

Den tisdagen den 23:e juli 2013 kl. 14:27:30 UTC+2 skrev Tommy Albinsson:

Hmms, is it that simple?

If one of the nodes goes down for example es1.domain it does not matter
then?

That would be just awesome if it was the case.

//Tommy

Den tisdagen den 23:e juli 2013 kl. 13:48:58 UTC+2 skrev Mark Walkom:

Just use a DNS CNAME that points to all 3 hosts.
ie elastic-cluster.domain = es1.domain, es2.domain, es3.domain.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 23 July 2013 21:32, Tommy Albinsson tommy.a...@gmail.com wrote:

Hi,

I am setting up a new environment with ElasticSearch, and the out of
the box zen discovery makes it real easy to load balance.

But if I want to set up my environment like the following.

Three servers running on the same network, works perfectly and they
find each other.

es1 10.2.88.21
es2 10.2.88.22
es3 10.2.88.23

But then, from my web application I want to access the cluster via ip
10.2.88.24, which is a virtual ip so that if one node goes down, it does
not matter since I always access 10.2.88.24 and it contains all three of
them.

Is my intentions clear? How would i configure ElasticSearch for this
purpose?

//Tommy

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.**com.
For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

If you use a cname the port doesn't matter. If a node goes down the the
connection will timeout, and your app should be able to gracefully handle
this and retry a connection which should connect to a different node. If
you keep the TTL for the cname down low (eg a minute), then you can easily
work around any extended downtime for a node by updating the record.
Using VIPs is good, but if you want something basic this might be the
easiest.

However a thought occurs that if you are using Windows, specifically server
2008, then you might be able to leverage Windows Clustering and have a VIP
and then point a cname to that. Windows Clustering is pretty easy too :slight_smile:

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 23 July 2013 23:12, Tommy Albinsson tommy.albinsson@gmail.com wrote:

I am on a windows environment also :slight_smile:

//Tommy

Den tisdagen den 23:e juli 2013 kl. 14:58:19 UTC+2 skrev Tommy Albinsson:

It does not have to be a virtual ip, I just want to use the same url to
access my cluster and regarding what node goes down it should not matter.

//Tommy

Den tisdagen den 23:e juli 2013 kl. 14:41:40 UTC+2 skrev Jörg Prante:

If you want a virtual IP, use ucarp. I use it with success, in front of
nginx, JBoss, httpd, whatever.

http://www.pureftpd.org/**project/ucarphttp://www.pureftpd.org/project/ucarp

Jörg

On Tue, Jul 23, 2013 at 2:31 PM, Tommy Albinsson tommy.a...@gmail.comwrote:

So it does not matter if ElasticSearch is on port 90200?

If I add elastic-cluster.domain as you describe and access it for
example via elast-cluster.domain:90200, that would be correct woldn't it?

//Tommy

Den tisdagen den 23:e juli 2013 kl. 14:27:30 UTC+2 skrev Tommy
Albinsson:

Hmms, is it that simple?

If one of the nodes goes down for example es1.domain it does not
matter then?

That would be just awesome if it was the case.

//Tommy

Den tisdagen den 23:e juli 2013 kl. 13:48:58 UTC+2 skrev Mark Walkom:

Just use a DNS CNAME that points to all 3 hosts.
ie elastic-cluster.domain = es1.domain, es2.domain, es3.domain.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 23 July 2013 21:32, Tommy Albinsson tommy.a...@gmail.com wrote:

Hi,

I am setting up a new environment with ElasticSearch, and the out of
the box zen discovery makes it real easy to load balance.

But if I want to set up my environment like the following.

Three servers running on the same network, works perfectly and they
find each other.

es1 10.2.88.21
es2 10.2.88.22
es3 10.2.88.23

But then, from my web application I want to access the cluster via
ip 10.2.88.24, which is a virtual ip so that if one node goes down, it does
not matter since I always access 10.2.88.24 and it contains all three of
them.

Is my intentions clear? How would i configure ElasticSearch for this
purpose?

//Tommy

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.**com.
For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

I would only suggest that the VIP is not used at all by elastic search
itself (I.e. TCP/9300) only use it to expose the cluster to external
entities.
I can imagine ES may get v confused if a node which was on one IP appeared
on another one without disconnecting from the cluster first... But maybe it
doesn't care?

BTW, why route all your requests through 1 IP only?
On 29/07/2013 6:51 PM, "Mark Walkom" markw@campaignmonitor.com wrote:

If you use a cname the port doesn't matter. If a node goes down the the
connection will timeout, and your app should be able to gracefully handle
this and retry a connection which should connect to a different node. If
you keep the TTL for the cname down low (eg a minute), then you can easily
work around any extended downtime for a node by updating the record.
Using VIPs is good, but if you want something basic this might be the
easiest.

However a thought occurs that if you are using Windows, specifically
server 2008, then you might be able to leverage Windows Clustering and have
a VIP and then point a cname to that. Windows Clustering is pretty easy too
:slight_smile:

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 23 July 2013 23:12, Tommy Albinsson tommy.albinsson@gmail.com wrote:

I am on a windows environment also :slight_smile:

//Tommy

Den tisdagen den 23:e juli 2013 kl. 14:58:19 UTC+2 skrev Tommy Albinsson:

It does not have to be a virtual ip, I just want to use the same url to
access my cluster and regarding what node goes down it should not matter.

//Tommy

Den tisdagen den 23:e juli 2013 kl. 14:41:40 UTC+2 skrev Jörg Prante:

If you want a virtual IP, use ucarp. I use it with success, in front of
nginx, JBoss, httpd, whatever.

http://www.pureftpd.org/**project/ucarphttp://www.pureftpd.org/project/ucarp

Jörg

On Tue, Jul 23, 2013 at 2:31 PM, Tommy Albinsson tommy.a...@gmail.comwrote:

So it does not matter if ElasticSearch is on port 90200?

If I add elastic-cluster.domain as you describe and access it for
example via elast-cluster.domain:90200, that would be correct woldn't it?

//Tommy

Den tisdagen den 23:e juli 2013 kl. 14:27:30 UTC+2 skrev Tommy
Albinsson:

Hmms, is it that simple?

If one of the nodes goes down for example es1.domain it does not
matter then?

That would be just awesome if it was the case.

//Tommy

Den tisdagen den 23:e juli 2013 kl. 13:48:58 UTC+2 skrev Mark Walkom:

Just use a DNS CNAME that points to all 3 hosts.
ie elastic-cluster.domain = es1.domain, es2.domain, es3.domain.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 23 July 2013 21:32, Tommy Albinsson tommy.a...@gmail.com wrote:

Hi,

I am setting up a new environment with ElasticSearch, and the out
of the box zen discovery makes it real easy to load balance.

But if I want to set up my environment like the following.

Three servers running on the same network, works perfectly and they
find each other.

es1 10.2.88.21
es2 10.2.88.22
es3 10.2.88.23

But then, from my web application I want to access the cluster via
ip 10.2.88.24, which is a virtual ip so that if one node goes down, it does
not matter since I always access 10.2.88.24 and it contains all three of
them.

Is my intentions clear? How would i configure ElasticSearch for
this purpose?

//Tommy

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/grou
ps/opt_out https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.**com.
For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

In that case, take nginx, and put it up in load balancing mode like
described here, with a short fail timeout

DNS round robin is not recommended, unless you accept downtimes for many
minutes. You can't control the TTL how it propagates through all the DNS
services when a server goes down.

Do not expect high performance and scalability with Windows.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

+1 on all counts ( wont comment on windows :wink: ) .

fwiw, our usual setup is
[ a pair of LBs using haproxy - http checks if using http, or simple tcp
if relying on thrift] --> [ a number n of instances running in master-only
mode, data= false ,effectively master + routing only] --> [ a number m of
data nodes].

Linux on EC2.

( got to play a bit with using specific http queries to get an http ok/
error from ES ,when using thrift ).

b

On Mon, Jul 29, 2013 at 9:15 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

In that case, take nginx, and put it up in load balancing mode like
described here, with a short fail timeout
https://www.digitalocean.com/community/articles/how-to-set-up-nginx-load-balancing

DNS round robin is not recommended, unless you accept downtimes for many
minutes. You can't control the TTL how it propagates through all the DNS
services when a server goes down.

Do not expect high performance and scalability with Windows.

Jörg

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Norberto 'Beto' Meijome

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

What about using an additional elasticsearch node (data=false,
master=false) instead of nginx?

Cheers

Am Montag, 29. Juli 2013 13:15:21 UTC+2 schrieb Jörg Prante:

In that case, take nginx, and put it up in load balancing mode like
described here, with a short fail timeout
https://www.digitalocean.com/community/articles/how-to-set-up-nginx-load-balancing

DNS round robin is not recommended, unless you accept downtimes for many
minutes. You can't control the TTL how it propagates through all the DNS
services when a server goes down.

Do not expect high performance and scalability with Windows.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

nginx is preferable if you want also request / response filtering, static
file serving, authentication, compression, logging, traffic shaping, http
flooding prevention, awesome concurrency. In combination with ucarp on a
local network, I set up nginx's in a row of machines, avoiding any SPOF
(within a data center) with failover of 1 second, no matter if there is a
Java EE JVM, or Perl/Python/Ruby/PHP processes connected to ES in the back
yard.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

@Jorg, have u tried doing tcp proxying w/nginx?
https://github.com/yaoweibin/nginx_tcp_proxy_module ( for use w/ ES-thrift
plugin)

On Tue, Jul 30, 2013 at 9:13 AM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

nginx is preferable if you want also request / response filtering, static
file serving, authentication, compression, logging, traffic shaping, http
flooding prevention, awesome concurrency. In combination with ucarp on a
local network, I set up nginx's in a row of machines, avoiding any SPOF
(within a data center) with failover of 1 second, no matter if there is a
Java EE JVM, or Perl/Python/Ruby/PHP processes connected to ES in the back
yard.

Jörg

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Norberto 'Beto' Meijome

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

No. What I'd like to use is nginx websocket proxying (for event processing
with ES, not implemented yet) http://nginx.org/en/docs/http/websocket.html

Jörg

On Tue, Jul 30, 2013 at 1:08 PM, Norberto Meijome numard@gmail.com wrote:

@Jorg, have u tried doing tcp proxying w/nginx?
https://github.com/yaoweibin/nginx_tcp_proxy_module ( for use w/
ES-thrift plugin)

On Tue, Jul 30, 2013 at 9:13 AM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

nginx is preferable if you want also request / response filtering, static
file serving, authentication, compression, logging, traffic shaping, http
flooding prevention, awesome concurrency. In combination with ucarp on a
local network, I set up nginx's in a row of machines, avoiding any SPOF
(within a data center) with failover of 1 second, no matter if there is a
Java EE JVM, or Perl/Python/Ruby/PHP processes connected to ES in the back
yard.

Jörg

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Norberto 'Beto' Meijome

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

@Norberto

Why route everythin to 1 ip only, that is because i don't want to target a
specific node because what happens if that ones goes down.

Ofcourse I can build this into my application to try different ip addresses
if one does not answer but setting up a VIP pointing containing all the
node ips does seem like a good idea, doesn't it?

Den måndagen den 29:e juli 2013 kl. 10:55:47 UTC+2 skrev Norberto Meijome:

I would only suggest that the VIP is not used at all by elastic search
itself (I.e. TCP/9300) only use it to expose the cluster to external
entities.
I can imagine ES may get v confused if a node which was on one IP appeared
on another one without disconnecting from the cluster first... But maybe it
doesn't care?

BTW, why route all your requests through 1 IP only?
On 29/07/2013 6:51 PM, "Mark Walkom" <ma...@campaignmonitor.com<javascript:>>
wrote:

If you use a cname the port doesn't matter. If a node goes down the the
connection will timeout, and your app should be able to gracefully handle
this and retry a connection which should connect to a different node. If
you keep the TTL for the cname down low (eg a minute), then you can easily
work around any extended downtime for a node by updating the record.
Using VIPs is good, but if you want something basic this might be the
easiest.

However a thought occurs that if you are using Windows, specifically
server 2008, then you might be able to leverage Windows Clustering and have
a VIP and then point a cname to that. Windows Clustering is pretty easy too
:slight_smile:

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com <javascript:>
web: www.campaignmonitor.com

On 23 July 2013 23:12, Tommy Albinsson <tommy.a...@gmail.com<javascript:>

wrote:

I am on a windows environment also :slight_smile:

//Tommy

Den tisdagen den 23:e juli 2013 kl. 14:58:19 UTC+2 skrev Tommy Albinsson:

It does not have to be a virtual ip, I just want to use the same url to
access my cluster and regarding what node goes down it should not matter.

//Tommy

Den tisdagen den 23:e juli 2013 kl. 14:41:40 UTC+2 skrev Jörg Prante:

If you want a virtual IP, use ucarp. I use it with success, in front
of nginx, JBoss, httpd, whatever.

http://www.pureftpd.org/**project/ucarphttp://www.pureftpd.org/project/ucarp

Jörg

On Tue, Jul 23, 2013 at 2:31 PM, Tommy Albinsson <tommy.a...@gmail.com

wrote:

So it does not matter if ElasticSearch is on port 90200?

If I add elastic-cluster.domain as you describe and access it for
example via elast-cluster.domain:90200, that would be correct woldn't it?

//Tommy

Den tisdagen den 23:e juli 2013 kl. 14:27:30 UTC+2 skrev Tommy
Albinsson:

Hmms, is it that simple?

If one of the nodes goes down for example es1.domain it does not
matter then?

That would be just awesome if it was the case.

//Tommy

Den tisdagen den 23:e juli 2013 kl. 13:48:58 UTC+2 skrev Mark Walkom:

Just use a DNS CNAME that points to all 3 hosts.
ie elastic-cluster.domain = es1.domain, es2.domain, es3.domain.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 23 July 2013 21:32, Tommy Albinsson tommy.a...@gmail.comwrote:

Hi,

I am setting up a new environment with ElasticSearch, and the out
of the box zen discovery makes it real easy to load balance.

But if I want to set up my environment like the following.

Three servers running on the same network, works perfectly and
they find each other.

es1 10.2.88.21
es2 10.2.88.22
es3 10.2.88.23

But then, from my web application I want to access the cluster via
ip 10.2.88.24, which is a virtual ip so that if one node goes down, it does
not matter since I always access 10.2.88.24 and it contains all three of
them.

Is my intentions clear? How would i configure ElasticSearch for
this purpose?

//Tommy

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/grou
ps/opt_out https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.**com.
For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

sure, u point your load balancer to a number of nodes, and your app speaks
to your LB(s) . pretty standard design...

On Wed, Jul 31, 2013 at 5:58 PM, Tommy Albinsson
tommy.albinsson@gmail.comwrote:

@Norberto

Why route everythin to 1 ip only, that is because i don't want to target a
specific node because what happens if that ones goes down.

Ofcourse I can build this into my application to try different ip
addresses if one does not answer but setting up a VIP pointing containing
all the node ips does seem like a good idea, doesn't it?

Den måndagen den 29:e juli 2013 kl. 10:55:47 UTC+2 skrev Norberto Meijome:

I would only suggest that the VIP is not used at all by elastic search
itself (I.e. TCP/9300) only use it to expose the cluster to external
entities.
I can imagine ES may get v confused if a node which was on one IP
appeared on another one without disconnecting from the cluster first... But
maybe it doesn't care?

BTW, why route all your requests through 1 IP only?
On 29/07/2013 6:51 PM, "Mark Walkom" ma...@campaignmonitor.com wrote:

If you use a cname the port doesn't matter. If a node goes down the the
connection will timeout, and your app should be able to gracefully handle
this and retry a connection which should connect to a different node. If
you keep the TTL for the cname down low (eg a minute), then you can easily
work around any extended downtime for a node by updating the record.
Using VIPs is good, but if you want something basic this might be the
easiest.

However a thought occurs that if you are using Windows, specifically
server 2008, then you might be able to leverage Windows Clustering and have
a VIP and then point a cname to that. Windows Clustering is pretty easy too
:slight_smile:

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 23 July 2013 23:12, Tommy Albinsson tommy.a...@gmail.com wrote:

I am on a windows environment also :slight_smile:

//Tommy

Den tisdagen den 23:e juli 2013 kl. 14:58:19 UTC+2 skrev Tommy
Albinsson:

It does not have to be a virtual ip, I just want to use the same url
to access my cluster and regarding what node goes down it should not matter.

//Tommy

Den tisdagen den 23:e juli 2013 kl. 14:41:40 UTC+2 skrev Jörg Prante:

If you want a virtual IP, use ucarp. I use it with success, in front
of nginx, JBoss, httpd, whatever.

http://www.pureftpd.org/project/ucarphttp://www.pureftpd.org/project/ucarp

Jörg

On Tue, Jul 23, 2013 at 2:31 PM, Tommy Albinsson <
tommy.a...@gmail.com> wrote:

So it does not matter if ElasticSearch is on port 90200?

If I add elastic-cluster.domain as you describe and access it for
example via elast-cluster.domain:90200, that would be correct woldn't it?

//Tommy

Den tisdagen den 23:e juli 2013 kl. 14:27:30 UTC+2 skrev Tommy
Albinsson:

Hmms, is it that simple?

If one of the nodes goes down for example es1.domain it does not
matter then?

That would be just awesome if it was the case.

//Tommy

Den tisdagen den 23:e juli 2013 kl. 13:48:58 UTC+2 skrev Mark
Walkom:

Just use a DNS CNAME that points to all 3 hosts.
ie elastic-cluster.domain = es1.domain, es2.domain, es3.domain.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 23 July 2013 21:32, Tommy Albinsson tommy.a...@gmail.comwrote:

Hi,

I am setting up a new environment with ElasticSearch, and the out
of the box zen discovery makes it real easy to load balance.

But if I want to set up my environment like the following.

Three servers running on the same network, works perfectly and
they find each other.

es1 10.2.88.21
es2 10.2.88.22
es3 10.2.88.23

But then, from my web application I want to access the cluster
via ip 10.2.88.24, which is a virtual ip so that if one node goes down, it
does not matter since I always access 10.2.88.24 and it contains all three
of them.

Is my intentions clear? How would i configure ElasticSearch for
this purpose?

//Tommy

--
You received this message because you are subscribed to the
Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.**c**om.
For more options, visit https://groups.google.com/grou

ps/opt_out https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.com.
For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.com.
For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Norberto 'Beto' Meijome

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.