ELK Architecture and requirements

Elastic Stack Elasticsearch
1

Hi guys,

I need to create an ELK architecture but I don't know how many servers and requirements (CPU, RAM,Disk space) I will need.
I will need to send syslog and logfiles from 15 servers (total around 500MB/day) to this ELK and have almost 60 days retention.
I also will need to separate in 3 tiers (web - kibana, app - logstash, db - elasticsearch) and I also want high availability.
So if the operating system is Red Hat Enterprise Linux 7 what should be the correct architecture?

Thanks,
Nelson

1 Like
2

You want at least 3 nodes, and that should hold your expected dataset.

Also we’ve renamed ELK to the Elastic Stack, otherwise Beats feels left out :wink:

1 Like
3

For that load pretty much any 3 modern machines will do fine. As for requirements more is more :slight_smile:

RAM:
A machine with 64 GB of RAM is the ideal sweet spot, but 32 GB and 16 GB machines are also common. Less than 8 GB tends to be counterproductive...

CPUs:
Most Elasticsearch deployments tend to be rather light on CPU requirements. As such, the exact processor setup matters less than the other resources. You should choose a modern processor with multiple cores. Common clusters utilize two- to eight-core machines.

Disk:
If you can afford SSDs, they are by far superior to any spinning media. SSD-backed nodes see boosts in both query and indexing performance....

Even with 1 TB of disk per node you would have 3TB / 2 (if you use one replica per shard) for your indices. 1500GB (total storage) /0.5 GB (daily data) = 3000 days of retention :stuck_out_tongue:

1 Like
4

Hi Mark,

What do you mean with 3 nodes? 1 node for each tier??
And what about 2 servers for logstash, 2 servers for kibana and 3 server for elasticsearch?
For elasticsearch, how many master, ingest and data nodes I need?

Thanks,
Nelson Pita

5

Hi,

How many servers by tier? What about 2 servers for logstash, 2 servers for kibana and 3 server for elasticsearch?
For elasticsearch, how many master, ingest and data nodes I need?

Thanks,
Nelson Pita

6

If you want your Elasticsearch cluster to be highly available, you need 3 nodes just for Elasticsearch. This assumes nodes that have the default configuration (master eligible, holds data and supports ingest if you need it).

Add to that nodes for Logstash and Kibana, probably 2 each if you need to separate out tiers.

1 Like
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.