How does loadbalancing and "talking to a cluster" work?

Elastic Stack Elasticsearch
1

Hi,
I have learned a lot about clusteres in the past 2 weeks but one thing which I dont get the hang of is how to talk to the cluster. What i mean is that I have a Redis Database and I would like to ingest data into a Hot-Warm cluster through Logstash. I know that a thing like an Ingest and Coordination Node exists but this would lead to single point of failures right?

grafik
grafik2067×625 53.3 KB

How do I set up a reliable cluster with no single point of failures (Failover procedures)? Is it possible to talk to all Nodes at once and the nodes itself decide which node has to perform which task? Hope you understand what I am trying to say.
Thanks
Defalt

1 Like
2

How much money do you have? I am not being flippant, it's an honest question as to how much you are willing to mitigate SPOFs.

No, you can talk to any node and it'll figure out where it needs to go. But that connection you have to that node is still a SPOF.

3

The default configuration has all nodes have all roles, and this is a great place to start. Just because you CAN have dedicated node types does not mean you SHOULD.

4

How much money do you have?

Around 5k-7k€. We think we would start with 3-5 Nodes and upgrade the cluster as we need more capacity.

Is it possible with logstash that it switches nodes if

  1. One node is under heavy load
  2. The node which its ingesting to goes down.
    I know that Kibana has such capabilites with elasticsearch.hosts:

My new idea for the cluster would be that Kibana and logstash have connections to all nodes and than they decide which one should be used(Just like elasticsearch.hosts. Is that the right approach?

5

In all honesty, I wouldn't be too worried about SPOFs. If this is business critical and you cannot do without downtime, then the business needs to value it appropriately.

Just aim for a 3+ node cluster, across different data centres in the same geographic region. Use replicas, automatic backups, stay up to date with your packages. Use ILM + SLM if you can. Things like that will help while maintaining your budget.

You may be even better off using something like Elasticsearch Service: Managed Elasticsearch on AWS, Google Cloud, and Azure | Elastic to reduce complexity, while getting access to all of the features - and more - that I mentioned.

The docs go into how it handles things on that level. But on a high level, yes.

6

Thanks for your great answers.

What are the best backup capabilities in ealastic? Is taking snapshots still pracitical or are there better backup technologies.

7

Snapshots are the best for backups. If you had the money you could run multiple clusters and then use cross cluster replication.

But even then, taking snapshots is a backup. CCR is not.

8

We currently have 8 Snapshots over the course of 4 years. What is the best way to restore all that data and how should we approach this in the future? Currently we are just restoring the snapshots and creating new Indices for each. Each snapshot contains 5 Indices so in the end we will have 40 indices.

9

That's not a lot. Our Elasticsearch Service takes them every 30 minutes.

Not sure what you mean by best way though. A restore is a restore.

10

Yes, but the restored data is smaller than the original index because its incremental. So is it better to reindex them together into one big index or multiple smaller ones.

Reading all your answers this would be the setup with which i came up with. Is this a good approach?

grafik
grafik1664×1053 55.7 KB

Thanks again for your help. :slight_smile:

11

Snapshots are incremental, so take them every N minutes, where N is your disaster recovery requirements.

12

A colleague raised the question if its possible to have Cluster A with online primary shards and indices and all the replicas are in the replica cluster B. Does this concept have any usecase?

13

Dear Mark, this is a very good point - what if you take a snapshot lets say every day from the Hot-node and delete all data which is older than 2 weeks. If you want to restore all data from an index (or at least 1 year or so) how can this be done since the last snapshot has not all data available right?

14

That's not possible, no.

15

Please see the delete section of https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshots-take-snapshot.html

16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.