Architecture Design

Elastic Stack Elasticsearch
1

Hey guys!!

I'm trying to understand how to scale my ELK setup to a production deployment... But first some specs on my current system:

  • Multiple Servers where I'm fetching data with Filebeat (around 10gb/day of logs)
  • One central server with Elasticsearch, Kibana and Logstash with: 32GB RAM, 500GB memory disk, 4 processing cores;

So basically I just have one node on my system.

But in order to scale it up to A LOT of more servers I'm trying to understand what kind of hardware requirements I need, for:

  • 100GB/day of log files
  • Dashboards usage
  • Retention period of two months

From the reading I've done all over the internet, I would have:

  • 3 Master Nodes
  • 3 Data Nodes, for storing all that data

Does this make any sense?? What kind of hardware requirements do I need in each node?? Also, in this case, each node means a server/VM, right??

Thanks a lot! All help would be highly appreciated... Just trying to understand this hardware area of the Elastic Stack, and it has not been easy!!

2

May I suggest you look at the following resources about sizing:

https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing

And https://www.elastic.co/webinars/using-rally-to-get-your-elasticsearch-cluster-size-right

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.