# How does loadbalancing and "talking to a cluster" work? **URL:** \ **Category:** Elasticsearch\ **Created:** [June 16, 2020, 8:18am UTC](https://discuss.elastic.co/t/how-does-loadbalancing-and-talking-to-a-cluster-work/237253 "2020-06-16T08:18:52Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![defalt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/defalt/32/71379_2.png) [@defalt](https://discuss.elastic.co/u/defalt)\ **Post date:** [June 16, 2020, 8:18am UTC](https://discuss.elastic.co/t/how-does-loadbalancing-and-talking-to-a-cluster-work/237253/1 "2020-06-16T08:18:52Z")
Hi, I have learned a lot about clusteres in the past 2 weeks but one thing which I dont get the hang of is how to talk to the cluster. What i mean is that I have a Redis Database and I would like to ingest data into a Hot-Warm cluster through Logstash. I know that a thing like an Ingest and Coordination Node exists but this would lead to single point of failures right? ![grafik](https://us1.discourse-cdn.com/elastic/original/3X/2/9/29162a9feb6e0cd199d9984d20d71bd34a33f96f.png) How do I set up a reliable cluster with no single point of failures (Failover procedures)? Is it possible to talk to all Nodes at once and the nodes itself decide which node has to perform which task? Hope you understand what I am trying to say. Thanks Defalt ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 16, 2020, 8:21am UTC](https://discuss.elastic.co/t/how-does-loadbalancing-and-talking-to-a-cluster-work/237253/2 "2020-06-16T08:21:58Z")
> [@defalt](#): > > How do I set up a reliable cluster with no single point of failures How much money do you have? I am not being flippant, it's an honest question as to how much you are willing to mitigate SPOFs. > [@defalt](#): > > Is it possible to talk to all Nodes at once and the nodes itself decide which node has to perform which task? No, you can talk to _any_ node and it'll figure out where it needs to go. But that connection you have to that node is still a SPOF. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 16, 2020, 8:24am UTC](https://discuss.elastic.co/t/how-does-loadbalancing-and-talking-to-a-cluster-work/237253/3 "2020-06-16T08:24:03Z")
The default configuration has all nodes have all roles, and this is a great place to start. Just because you CAN have dedicated node types does not mean you SHOULD. ---
**Author:** ![defalt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/defalt/32/71379_2.png) [@defalt](https://discuss.elastic.co/u/defalt)\ **Post date:** [June 16, 2020, 8:29am UTC](https://discuss.elastic.co/t/how-does-loadbalancing-and-talking-to-a-cluster-work/237253/4 "2020-06-16T08:29:55Z")
> How much money do you have? Around 5k-7k€. We think we would start with 3-5 Nodes and upgrade the cluster as we need more capacity. > [@warkolm](#): > > No, you can talk to _any_ node and it'll figure out where it needs to go. Is it possible with logstash that it switches nodes if 1. One node is under heavy load 2. The node which its ingesting to goes down. I know that Kibana has such capabilites with `elasticsearch.hosts:` > [@Christian\_Dahlqvist](#): > > all nodes have all roles My new idea for the cluster would be that Kibana and logstash have connections to all nodes and than they decide which one should be used(Just like `elasticsearch.hosts`. Is that the right approach? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 16, 2020, 8:38am UTC](https://discuss.elastic.co/t/how-does-loadbalancing-and-talking-to-a-cluster-work/237253/5 "2020-06-16T08:38:43Z")
> [@defalt](#): > > Around 5k-7k€. We think we would start with 3-5 Nodes and upgrade the cluster as we need more capacity. In all honesty, I wouldn't be too worried about SPOFs. If this is business critical and you cannot do without downtime, then the business needs to value it appropriately. Just aim for a 3+ node cluster, across different data centres in the same geographic region. Use replicas, automatic backups, stay up to date with your packages. Use ILM + SLM if you can. Things like that will help while maintaining your budget. You may be even better off using something like [Elasticsearch Service: Managed Elasticsearch on AWS, Google Cloud, and Azure | Elastic](https://www.elastic.co/elasticsearch/service) to reduce complexity, while getting access to all of the features - and more - that I mentioned. > [@defalt](#): > > Is it possible with logstash that it switches nodes if [The docs](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html) go into how it handles things on that level. But on a high level, yes. ---
**Author:** ![defalt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/defalt/32/71379_2.png) [@defalt](https://discuss.elastic.co/u/defalt)\ **Post date:** [June 16, 2020, 8:44am UTC](https://discuss.elastic.co/t/how-does-loadbalancing-and-talking-to-a-cluster-work/237253/6 "2020-06-16T08:44:54Z")
Thanks for your great answers. > [@warkolm](#): > > automatic backups What are the best backup capabilities in ealastic? Is taking snapshots still pracitical or are there better backup technologies. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 16, 2020, 8:45am UTC](https://discuss.elastic.co/t/how-does-loadbalancing-and-talking-to-a-cluster-work/237253/7 "2020-06-16T08:45:56Z")
Snapshots are the best for backups. If you had the money you could run multiple clusters and then use cross cluster replication. But even then, taking snapshots is a backup. CCR is not. ---
**Author:** ![defalt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/defalt/32/71379_2.png) [@defalt](https://discuss.elastic.co/u/defalt)\ **Post date:** [June 16, 2020, 8:48am UTC](https://discuss.elastic.co/t/how-does-loadbalancing-and-talking-to-a-cluster-work/237253/8 "2020-06-16T08:48:24Z")
We currently have 8 Snapshots over the course of 4 years. What is the best way to restore all that data and how should we approach this in the future? Currently we are just restoring the snapshots and creating new Indices for each. Each snapshot contains 5 Indices so in the end we will have 40 indices. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 16, 2020, 8:51am UTC](https://discuss.elastic.co/t/how-does-loadbalancing-and-talking-to-a-cluster-work/237253/9 "2020-06-16T08:51:26Z")
That's not a lot. Our Elasticsearch Service takes them every 30 minutes. Not sure what you mean by best way though. A restore is a restore. ---
**Author:** ![defalt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/defalt/32/71379_2.png) [@defalt](https://discuss.elastic.co/u/defalt)\ **Post date:** [June 16, 2020, 8:57am UTC](https://discuss.elastic.co/t/how-does-loadbalancing-and-talking-to-a-cluster-work/237253/10 "2020-06-16T08:57:32Z")
> [@warkolm](#): > > A restore is a restore. Yes, but the restored data is smaller than the original index because its incremental. So is it better to reindex them together into one big index or multiple smaller ones. Reading all your answers this would be the setup with which i came up with. Is this a good approach? ![grafik](https://us1.discourse-cdn.com/elastic/original/3X/8/e/8e06430fdb96f288a36369bf94184bc5f9438dea.png) Thanks again for your help. 🙂 ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 16, 2020, 9:13am UTC](https://discuss.elastic.co/t/how-does-loadbalancing-and-talking-to-a-cluster-work/237253/11 "2020-06-16T09:13:30Z")
Snapshots are incremental, so take them every N minutes, where N is your disaster recovery requirements. ---
**Author:** ![defalt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/defalt/32/71379_2.png) [@defalt](https://discuss.elastic.co/u/defalt)\ **Post date:** [June 16, 2020, 9:20am UTC](https://discuss.elastic.co/t/how-does-loadbalancing-and-talking-to-a-cluster-work/237253/12 "2020-06-16T09:20:43Z")
A colleague raised the question if its possible to have Cluster A with online primary shards and indices and all the replicas are in the replica cluster B. Does this concept have any usecase? ---
**Author:** ![azeiner](https://avatars.discourse-cdn.com/v4/letter/a/e19b73/32.png) [@azeiner](https://discuss.elastic.co/u/azeiner)\ **Post date:** [June 16, 2020, 2:05pm UTC](https://discuss.elastic.co/t/how-does-loadbalancing-and-talking-to-a-cluster-work/237253/13 "2020-06-16T14:05:51Z")
Dear Mark, this is a very good point - what if you take a snapshot lets say every day from the Hot-node and delete all data which is older than 2 weeks. If you want to restore all data from an index (or at least 1 year or so) how can this be done since the last snapshot has not all data available right? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 16, 2020, 10:47pm UTC](https://discuss.elastic.co/t/how-does-loadbalancing-and-talking-to-a-cluster-work/237253/14 "2020-06-16T22:47:22Z")
That's not possible, no. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 16, 2020, 10:48pm UTC](https://discuss.elastic.co/t/how-does-loadbalancing-and-talking-to-a-cluster-work/237253/15 "2020-06-16T22:48:31Z")
Please see the delete section of [https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshots-take-snapshot.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshots-take-snapshot.html) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 14, 2020, 10:48pm UTC](https://discuss.elastic.co/t/how-does-loadbalancing-and-talking-to-a-cluster-work/237253/16 "2020-07-14T22:48:40Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.