Benchmarking cluster with rally

[Abdelhalim]

Hello,

I have set a development cluster with 3 master Nodes, 2 hot data nodes and 1 warm node, and I wanted to benchmark it to reproduce results which looks like the results got in here : Benchmarking and sizing your Elasticsearch cluster for logs and metrics | Elastic Blog

so I tried to benchmark with this command :

 esrally race --track=metricbeat --target-hosts=10.13.81.11:9200,10.13.81.12:9200,10.13.81.13:9200,10.13.81.21:9200,10.13.81.22:9200,10.13.81.23:9200 --pipeline=benchmark-only --client-options='use_ssl:true,verify_cert:true,ca_certs:'ca.crt',client_cert:'beat.crt',client_key:'beat.key',basic_auth_user:'elastic',basic_auth_password:'MyPassword'' --track-params='number_of_shards:1' --track-params='number_of_replicas:0'

I tested 3 times, with 1 shard, 2 shard and 3 shard. and I always have the same result of the mean Throughput which equal to : 14.9K/s

I also tested in one node with the command :

esrally race --distribution-version=7.13.0 --track=metricbeat

and i got 11K/s

Could you tell me please why I am not seeing a big difference ? and what I am doing wrong if it's not the expected result as I think

Best regards

[dliappis (Dimitrios Liappis)]

Assuming that shard settings got applied correctly here are a few thoughts.

The workload that you've chosen (metricbeat) is quite small and very compressible. What was the duration of your benchmark? It's very likely it was very short.

For evaluating indexing throughput, you probably want to use something that resembles more the workload that you are after and is larger. You can e.g. the http_logs track for something more logs oriented, pmc if you are after large docs or nyc_taxis for a rather large corpora. For most tracks there are README pages describing the workload e.g. rally-tracks/http_logs at master · elastic/rally-tracks · GitHub

You could also be having a bottleneck elsewhere, e.g. on your loaddriver, or elsewhere.

I recommend taking a systematic approach ensuring that you are running valid benchmarks and detecting where the bottleneck is; there is a great tutorial linked in the Rally docs page that you can find here: The Seven Deadly Sins of Elasticsearch Benchmarking | Elastic or here: Benchmarking Elasticsearch with Rally by Daniel Mitterdorfer | Search Meetup Munich - YouTube

[dliappis (Dimitrios Liappis)]

With all the caveats mentioned in the previous comment (small workload etc.) out of curiosity I tried the metricbeat track against:

1. 1-node environment, number_of_shards:0: 26863 docs/s
2. 3-node environment, number_of_shards:5: 39887.4 docs/s

and as expected got higher indexing throughput on the 3-node env.

So I'd definitely look at bottlenecks and issues with your benchmarking methodology.