Hi,
I am trying to benchmark ES cluster. The cluster has a single index with single shard. We need to benchmark no. of read operations supported by the cluster.
If we have 3 master eligible data nodes (all data nodes having one copy of index), the cluster gives approx 90 throughput (requests/sec). Here we have one primary and 2 replicas of the index. The index has single shard.
On adding one more data node to the cluster (which means 4 data node ES cluster) and setting replica count as 3 for the index, the cluster gives throughput approx 120 requests/sec. Kibana graph shows that all nodes have almost same load and CPU utilization.
On adding one more data node to the cluster (5 data nodes ES cluster) and setting replica count as 4 for the index, the cluster gives throughput approx 150 requests/sec. Kibana graph shows that all nodes have almost same load and CPU utilization.
Till this point, we are getting additional throughput of nearly 30 requests/sec on adding a node to the cluster.
When we add one more data node to the cluster (6 data nodes) and set replica count as 5 for the index, so that every node has a copy of the index, cluster throughput falls to approx 120 to 125 requests/sec. Also kibana graph shows that only one node is getting utilized more as compared to other nodes. Ideally the throughput served by 6 data nodes of same size should have been ~ 180 req/sec.
Here we are having each node with 2 CPUs and 8GB memory (RAM).
Can someone help in understanding this behavior?
Is there any other better way to scale the cluster?
We want to benchmark read requests as our index will be having more read operations compared to write operations.