How to use a load balancer with Elasticsearch?

Elastic Stack Elasticsearch
1

For load balancing, there are actually a few out of the box solutions available without having to use an external load balancer.

  • Elasticsearch also provides a light weight load balancer in the form of what we call a client node (http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-node.html#modules-node). This is a separate ES node (and will join your main cluster) that does not store any data and it is “smart”, meaning that it knows where the relevant data is stored, so it will automatically direct requests to the appropriate node(s) that store the data being requested. Once it receives the data back from the data nodes, the client node will reduce the result set and send back to the requestor.
  • If you are using the .NET client, you can use one of the available IConnectionPool types (eg. SniffingConnectionPool and StaticConnectionPool): http://nest.azurewebsites.net/elasticsearch-net/connecting.html and http://nest.azurewebsites.net/elasticsearch-net/cluster-failover.html. For example, in your application that calls ES.net, you can define a SniffingConnectionPool with a list of seed URLs for the nodes when you establish the ES connection. With a SniffingConnectionPool setting, the .NET client will sniff the cluster state to determine the list of "alive" nodes and round robin requests automatically between them.
  • If you are using the NEST client, you can similarly pass in the appropriate IConnectionPool type to your ConnectionSettings to achieve the same. This is a pretty straightforward way to implement load balancing type of feature in Elasticsearch.
1 Like
2

We had ES cluster in Azure where we were initially leveraging Azure's Internal Load Balancer but we moved away from that and use NEST now. It just works fine :smile:

3

I am interested in this. i am indexing lot of docs using NEST and reaching some limitations and connection refused by 1 node. If i have 2 es nodes is it better to put a linux loadbalancer in front of them or use NEST? Buf if using NEST you need to have 2 public ip address for the nodes right? if you just have one it is better to use the linux load balancing i guess.