How to calculate read/write throughput at cluster level?

Elastic Stack Elasticsearch
1

I am working on designing es cluster with below given setup

master nodes 3 (m5.a large) 2vcpu, 8gb
data nodes 2 (m5.ax large) 4vcpu, 16 gb

Keeping default elastic configuration currently.
read thread pool - 7
write thread pool -4

Calculation on throughput

Read
Data Node - 2
Threads per data node - 7
Shards - 2
Replica- 1
RPS - (2 * 7)/2 = (7 * 1000)/ 50 = 140 reqs/sec (Considering 50 ms response time)
RPM - 140 * 60 = 8400 reqs/sec

Write
Data Node - 2
Threads per data Node - 4
Shards - 2
RPS - (2 * 4)= 8 * (1000/20) = 400 reqs/sec (assuming 20 ms response time for write)
RPM = 400 * 60 = 24000

Please check these numbers, and let me know your suggestion.
Thanks.

2

There is no such thing as a single excel sheet where you type in the number of documents and get back the throughput. Everything is an approximation because of many different factors like document, mapping complexity, queries per second, query complexity, etc...

I would advise you to go ahead and benchmark your cluster with your own data. You can start simple with a single node, having a single shard and add data to it while querying it. See if it holds up your SLAs reagrding reading and writing. At some point you found out the sweet spot per shard and can start scaling based on nodes.

There is also a benchmarking tool for Elasticsearch called rally, you should take a look at.

Hope that helps as a start.

1 Like
3

Thanks a lot @spinscale . Was just trying to figure out number in terms of approximation.
Will explore given benchmarking tool.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.