I'm setting up elasticsearch in AWS and I'm trying to figure out how
to deal with a node going down. Specifically, if I have 2 ES servers
[s01, s02], and my app is configured to always hit s01, how can i get
it to failover to s02 if s01 goes down? My first thought was to put
ES behind an ELB which has health monitoring built in, but there is no
way to do a private internal only ELB so if I did this then ES would
be public to the world, which is clearly a bad idea. We are using
nodejs and I haven't yet found a library for it to do this failover
for me. I could write one I suppose but I'd rather avoid that if
possible.
I think there are 2 easy cases to distinguish:
1] Your app is external to AWS - in which case using a LB is fine (eg
I have AWS "pinging" ES via tomcat and use the LB's alerting and
failover etc etc)
2] Your app is internal to AWS - in this case, you create a "nodata"
node running on your app server, and it can then use the EC2 discovery
plugin (which I assume you already have installed - check it out if
not) to find all nodes and connect to a live one (ie you connect to
localhost:9300 instead of s01:9300)
Then there's the unpleasant 3rd case where you're running everything
internally within AWS but still have to access ES from the app via a
web server (etc) for security/networking reasons, in which case you
have to implement your own load balancer equivalent.
But it sounded like you were case [2], in which case the EC2 plugin
and a nodata node is an easy solution (and are well documented on the
main web page).
Hope this helps!
Yeah we're in situation 2. That looks like a pretty workable
solution, and really the same as what we're doing for mongodb with
mongos.
Thanks!
On Feb 9, 3:12 pm, Alex at Ikanow apigg...@ikanow.com wrote:
I think there are 2 easy cases to distinguish:
1] Your app is external to AWS - in which case using a LB is fine (eg
I have AWS "pinging" ES via tomcat and use the LB's alerting and
failover etc etc)2] Your app is internal to AWS - in this case, you create a "nodata"
node running on your app server, and it can then use the EC2 discovery
plugin (which I assume you already have installed - check it out if
not) to find all nodes and connect to a live one (ie you connect to
localhost:9300 instead of s01:9300)Then there's the unpleasant 3rd case where you're running everything
internally within AWS but still have to access ES from the app via a
web server (etc) for security/networking reasons, in which case you
have to implement your own load balancer equivalent.But it sounded like you were case [2], in which case the EC2 plugin
and a nodata node is an easy solution (and are well documented on the
main web page).Hope this helps!
Which library are you using? You mentioned NodeJS, so perhaps you are
working with plain javascript. AFAIK, client nodes (using port
9300/Thrift) are only possible using the Java library since every
other library uses the REST protocol. Someone please correct me if I
am wrong since I have only used the Java library with the occasional
curl usage.
If you are indeed using some library that works over REST, you will
need to implement fail-over within your application. A load balancer
would be the ideal solution.
--
Ivan
On Thu, Feb 9, 2012 at 12:46 PM, rbrc wednesday@gmail.com wrote:
Yeah we're in situation 2. That looks like a pretty workable
solution, and really the same as what we're doing for mongodb with
mongos.
Thanks!On Feb 9, 3:12 pm, Alex at Ikanow apigg...@ikanow.com wrote:
I think there are 2 easy cases to distinguish:
1] Your app is external to AWS - in which case using a LB is fine (eg
I have AWS "pinging" ES via tomcat and use the LB's alerting and
failover etc etc)2] Your app is internal to AWS - in this case, you create a "nodata"
node running on your app server, and it can then use the EC2 discovery
plugin (which I assume you already have installed - check it out if
not) to find all nodes and connect to a live one (ie you connect to
localhost:9300 instead of s01:9300)Then there's the unpleasant 3rd case where you're running everything
internally within AWS but still have to access ES from the app via a
web server (etc) for security/networking reasons, in which case you
have to implement your own load balancer equivalent.But it sounded like you were case [2], in which case the EC2 plugin
and a nodata node is an easy solution (and are well documented on the
main web page).Hope this helps!
Hi,
Doesn't option 2) create a SPOF? Or are you suggesting to run more
than 1 non-data ES node?
Otis
Search Analytics - Cloud Monitoring Tools & Services | Sematext
On Feb 9, 3:12 pm, Alex at Ikanow apigg...@ikanow.com wrote:
I think there are 2 easy cases to distinguish:
1] Your app is external to AWS - in which case using a LB is fine (eg
I have AWS "pinging" ES via tomcat and use the LB's alerting and
failover etc etc)2] Your app is internal to AWS - in this case, you create a "nodata"
node running on your app server, and it can then use the EC2 discovery
plugin (which I assume you already have installed - check it out if
not) to find all nodes and connect to a live one (ie you connect to
localhost:9300 instead of s01:9300)Then there's the unpleasant 3rd case where you're running everything
internally within AWS but still have to access ES from the app via a
web server (etc) for security/networking reasons, in which case you
have to implement your own load balancer equivalent.But it sounded like you were case [2], in which case the EC2 plugin
and a nodata node is an easy solution (and are well documented on the
main web page).Hope this helps!
Hi,
There are several options to handle client failover to the cluster.
The first, is that you can have several hosts listed on your client side and the client lib can handle the failover aspect. I am not sure if there is a nodejs lib that does that.
The second, mentioned in this thread, is to start an ES instance on the same box as the client (application), with a setting of node.client set to true in the config. In this case, the application talks to "localhost" over port 9200 (you are using HTTP, and not 9300 as was mentioned), and the local ES instance will do the discovery of the cluster for you (using ec2 discovery if you configured). It will be a smart client in this case, directing requests directly to the relevant nodes that the relevant shards exists on (no double hop).
Ivan: A standalone node can be configured with node.client set to true (its the same as setting node.data set to false, and node.master set to false). It just means that it will not hold any data, and won't ever become a master of the cluster.
Otis: This usage pattern is not a SPOF, since the client ES node is running with the application itself, and connects to the cluster as a client node. Assuming one has several of those instances running (to handle availability on the application layer, regardless of ES) you are good to go.
On Friday, February 10, 2012 at 8:06 PM, Otis Gospodnetic wrote:
Hi,
Doesn't option 2) create a SPOF? Or are you suggesting to run more
than 1 non-data ES node?Otis
Search Analytics - Cloud Monitoring Tools & Services | Sematext
On Feb 9, 3:12 pm, Alex at Ikanow <apigg...@ikanow.com (http://ikanow.com)> wrote:
I think there are 2 easy cases to distinguish:
1] Your app is external to AWS - in which case using a LB is fine (eg
I have AWS "pinging" ES via tomcat and use the LB's alerting and
failover etc etc)2] Your app is internal to AWS - in this case, you create a "nodata"
node running on your app server, and it can then use the EC2 discovery
plugin (which I assume you already have installed - check it out if
not) to find all nodes and connect to a live one (ie you connect to
localhost:9300 instead of s01:9300)Then there's the unpleasant 3rd case where you're running everything
internally within AWS but still have to access ES from the app via a
web server (etc) for security/networking reasons, in which case you
have to implement your own load balancer equivalent.But it sounded like you were case [2], in which case the EC2 plugin
and a nodata node is an easy solution (and are well documented on the
main web page).Hope this helps!
I have the following situation: I have a .NET client in clustered Windows
Servers in AWS. I want to deal with failover, but since my client runs in
Windows the solution of starting a ES node in the same server doesn't
apply. What's the best solution in this scenario?
I want to understand and know how to deal with the security issue related
to Elastic Load Balancer mentioned by rbrc, because one solution would be
to use ELB. If possible, I would like to understand more about the
following items and how one or some of them could help me:
- No-data server and EC2 discovery;
- Nginx authentication with the ELB solution;
- Using HAProxy instead of ELB.
Em quinta-feira, 9 de fevereiro de 2012 16h08min35s UTC-2, rbrc escreveu:
I'm setting up elasticsearch in AWS and I'm trying to figure out how
to deal with a node going down. Specifically, if I have 2 ES servers
[s01, s02], and my app is configured to always hit s01, how can i get
it to failover to s02 if s01 goes down? My first thought was to put
ES behind an ELB which has health monitoring built in, but there is no
way to do a private internal only ELB so if I did this then ES would
be public to the world, which is clearly a bad idea. We are using
nodejs and I haven't yet found a library for it to do this failover
for me. I could write one I suppose but I'd rather avoid that if
possible.