Elasticsearch sizing: Number of nodes, RAM needed

Elastic Stack Elasticsearch
1

I am trying to build an ES cluster to handle storing and searching event logs from security devices with following requirements: 25000 EPS, 1000 bytes/event, using hot/warm architecture with 4 days of hot and 90 days of warm. My question is how much ES nodes do I need? And how much RAM shoud I use? Thanks.

Variable Description Calculation Value Unit
EPS Events per second Estimate 25000 Events
AES Average event size Estimate 1000 Bytes
DUPD Disk used per day AESxEPSx60x60x24/1073741824 2011.656761 GB
miSPD Min Shards per day DUPD/50 40.23313522 Shards
HD Hot days Estimate 4 Days
WD Warm days Estimate 90 Days
xHD Expected data on hot days (for indexing, searching, forge merging) DUPD*HD/1024 7.858034223 TB
xWD Expected data on warm days (read only indices for searching and creating replicas) DUPD*WD 176.80577 TB
miHD Minimum disk size for hot days xHD*1.4 11.00124791 TB
miWD Minimum disk size for warm days xWD*1.4 247.528078 TB
2

This will depend on your data, hardware and query requirements. I would recommend looking at the following resources:

https://www.elastic.co/webinars/using-rally-to-get-your-elasticsearch-cluster-size-right

https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing

https://www.elastic.co/webinars/optimizing-storage-efficiency-in-elasticsearch

https://www.elastic.co/elasticon/conf/2018/sf/the-seven-deadly-sins-of-elasticsearch-benchmarking

1 Like
3

Thank you sir.
Very useful articles!

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.