HA-Cluster: Simple design on prem, self managed

Elastic Stack Elasticsearch
1

Hi there,
I am looking for a simple HA-design for a logging use case, on prem & self managed.
Would this be a good design?

NODE-1 master, data, ingest, kibana, logstash, redis
NODE-2 master, data, ingest, kibana, logstash, redis
NODE-3 master for quorum
NODE-4 fleet server

Or something like that perhaps?

image
image742Ă—747 8.67 KB

With each node in a different data center and 1 replica.
Any insights, thoughts?
Many thanks and kind regards!
Stefano

2

why not make node-3 same as node2 unless you have space issue. you will have three data node. incase you have one down replica will fail over.

also can use load balance for elastic:9200 and kibana:5600 hence even one down node will not cause user issue.

3

Indeed that would be beneficial. But the 3rd data center is just a very small one. So I cannot deploy a data node there.

kind regards
Stefano

4

Is this a hard requirement? Having a cluster with nodes in different datacenters would not be recommended as it may add a lot of issues, specially related to latency between nodes, I would not recommend doing this.

Also, where redis at? Are you using it with Logstash? Where would your Logstash and Kibana be located?

You described that you would want one elasicsearch node on each datacenter, but in this scenario where would Kibana and Logstash be?

5

It Depends™ but do note that this is how most clusters in Elastic Cloud run - each AZ in AWS/GCP/Azure/... is a different datacenter, separated by a few (tens of) kilometers from the other AZs in the same region.

There's some docs here on this subject:

You can also stretch your cluster across nearby data centers as long as the network interconnection between each pair of data centers is good enough.

They also go into quite some depth about the requirements for a HA cluster, which basically align with the design in the OP.

My only concern would be the single-point-of-failure Fleet Server instance. I don't know much about Fleet Server tho, so maybe this is ok?

6

With Fleet unavailable, the user will lost management over the agents enrolled on it, it will not impact data collection, but changes on policies will not be replicated until Fleet is back again.

But Fleet supports HA in a similar way as Kibana, so this is pretty easy to implement.

7

Hi Leandro, thanks for your reply!
You wrote: 'Is this a hard requirement? Having a cluster with nodes in different datacenters would not be recommended as it may add a lot of issues'

Actually I thought having a foot / server in 3 different data centers is a must have / best practice for real HA. I was thinking about: what happens if a data center goes down? Never mind, still have data in the other one.

Or am I wrong?
Kind regards,
Stefano

8

Hi David, thanks for you reply!
Actually these are private data centers beloging and managed by the entity who runs the elasicsearch cluster I am going to build. Meaning: no cloud, by on prem and self managed.
By the way: Do you think a 1 Gbit/s network is enough to connect one of the nodes to the other two? This with on 1Gbit/s is going to be the quorum master node.

Thanks and kind regards
Stefano

9

Indeed, as long as they behave like AZs in a single CSP region, you're not doing anything unusual.

We cannot say, it depends entirely on your workload and performance goals. You'll need to perform some experiments.

10

Thank you David. By the way could send me an Elastic contact (Germany, NRW). We would like to discuss the possibility of a licence? I was trying this with no luck for the last few days.

Thanks & Cheers
Stefano

11

The best way is to fill out the form at Have questions? Contact Elastic | Elastic

12

Let us presume I buy a licence then I have searcheable snapshots.
Starting from this design (3 nodes)

image
image451Ă—963 23.7 KB

How would it change if I want to use ILM & hot data & frozen data?
I would like to spend less for ssds and have logging data to hold for 90 to 180 days.
What would be the next best / simple design to add the searcheable snapshot technology?

Thanks for any input!
Stefano

13

You need disk space for the data you want to hold, since it would be frozen data, you do not need SSDs, normal HDDs would work.

There is also the option to use an object storage on cloud like S3 to store the frozen data, but for an on-premises deployment I'm not sure this is a good approach, I'm also not sure how the local s3 altenatives are working now since MinIO discontinued the community version.

If I'm not wrong, you also need to have the role data_frozen on the nodes so the cache for the snapshot data can be mounted.

In theory you could have the data_frozen on the hot nodes as well, but I'm not sure if this is recommended or if it will have any impact, I'm using Elastic Cloud and each data role is a different node.