ELK architecture

Elastic Stack Elasticsearch
1

Hello, I need some advice. I have an Elasticsearch cluster with six nodes (due to licensing). I have two locations, so ideally, I want to have data in pairs.

I collect data through
Filebeat -> Logstash(server) -> Elasticsearch
or through
Logstash(client) -> Logstash(server) -> Elasticsearch

I need to keep the data for two years. One month of data must be on SSD.

I expect around 2TB of data per month, which amounts to about 50TB over two years.

Could I have a design like this:

node.roles: master, data_hot
node.roles: master, data_hot

node.roles: data_warm, master-eligible
node.roles: data_warm, master-eligible

node.roles: data_cold
node.roles: data_cold

Where should I run the Logstash-server nodes, Kibana, and ML node or should I run it on different servers?

The CPU can be up to 10 on each server.
The RAM can be up to 64GB on each server.
DISK for 1: 3TB SSD?
DISK for 2: 10TB SSD?
DISK for 3: 50TB HDD?

And I think that I dont understand what Ingest node means, do I need some? I index data by Logstash.

Thanks.

2

Can you provide more context about this? You have a license for 6 nodes?

What do you mean with 2 locations? 2 different data centers? If so, this is not recommended and can lead to some issues.

Are those numbers taking into consideration the usage of replicas? If you want to have a resilient cluster you need to have at least one replicas.

Assuming that this is not considering replicas you would need then 4 TB per month and 100 TB total.

It is recommended to run Logstash and Kibana on separate nodes, ML nodes is a role that elasticsearch nodes can have, so you can set this role in your data nodes, keep in mind that ML nodes requires a license as well, if you have a license for 6 nodes it does not make much sense to have dedicated ML nodes.

Also, the main difference between hot and warm nodes are the hardware, so the cold tier does not make much sense on on-premises deployment.

One alternative would be to have 3 hot nodes and 3 warm + master nodes.

3

In addition to what Leandro said also be aware that you can not set up a highly available Elasticsearch cluster that can withstand the loss of a full location/availability zone across 2 availability zones or locations as a strict majority of master eligible nodes is always required to be present for a healthy cluster (3-out-of-3 or 3-out-of-4 etc). This is described in greater detail in the official documentation. You are however able to have resilience against single node failure even with 2 locations.

4

I have platinum license for 6 nodes. I have 2 production servers in different data centers both with 50 CPU and 400 GB RAM.
I have on-premise solution.

No these are numbers without replica. I want to have one replica. So I am ready to buy 3TB SSD disks to both and 50 TB HDD to both.

Okay I understand. I will run logstash and kibana on different nodes. I will run ML on hot nodes, because I think they will use these data.

I understand but I have only 2 production servers.

I can run:
1)
2 master nodes
2hot+ML+master-eligible
2warm

  • unlicensed 2logstash + 2kibana

or

2master+hot+ML
2warm+master-eligible
2cold

  • 2logstash + 2kibana

am I right? what will be better? Can you give me some tips?

5

With just 6 licenses I see no reason to have dedicated master nodes and there is also no reason to have cold nodes, so you should use hot and warm nodes.

Hot nodes would have SSD disks and warm nodes would haver HDD disks.

So you could have something like this:

  • 2 hot + ml + ingest + master eligible nodes
  • 4 warm (you will need at least one of them to be a master-eligible)

Kibana and Logstash depends on your requirements, you could have one per datacenter, they do not count against the license.

Personally I would not use this in production because having a cluster accross different datacenters can lead to many issues, but since you have this limitation this configuration seems to be a good approach.

2 Likes
6

Okay thanks a lot.
I didnt know I dont need to have special master nodes.
Yes that's what customer owns and I will try my best to fulfill his expectations.