Rally Benchmark - Which race/benchmark to use for performance testing

[mmadhana (Murali Madhanagopal)]

Hi, I am using Rally on ECK to test the performance of my hardware - CPU cores/memory, storage, and network usage. Which benchmark/race is compute-intensive and has a mix of indexing and concurrent search workload? I used NYC Taxis, not sure if it is the right test. Could you please help?

[Bradley_Deam]

Hi @mmadhana - apologies for the delayed response here!

Which benchmark/race is compute-intensive and has a mix of indexing and concurrent search workload? I used NYC Taxis, not sure if it is the right test. Could you please help?

We actually recently added concurrent indexing & querying to both the NYC Taxis and PMC tracks, so you can use either of these, however with benchmarking it's usually preferred to identify a track that represents your intended production workload as closely as possible, or create your own custom track if there's nothing already available.

If you have workloads that use document models/sizes and queries that are similar to that of the NYC Taxis track, then that's great! If not, then my suggestion would be to create a custom track using data that is as close to your workload as possible (note you can create a track from data in an existing cluster using Rally), that way you can get a more accurate idea of system resource utilisation with data that is close to what will be seen in a production environment, rather than running an arbitrary benchmarking experiment that is not likely to give you accurate ideas of projected throughput and latencies.

I'd also like to add that if you do want to try out the NYC Taxis track (or any other, for that matter) and find that your hardware isn't being fully utilised, you should look at adjusting the bulk_size, bulk_indexing_clients, and/or the number_of_shards under index_settings to something that makes sense for your cluster - e.g. you may want to set number_of_shards equal to the number of nodes in the benchmark target cluster.

Lastly, I'd strongly recommend taking a look at this ElasticON presentation on benchmarking Elasticsearch: " The Seven Deadly Sins of Elasticsearch Benchmarking" - that'll make sure you don't run into any of the common pitfalls made when attempting to run benchmarks.

I hope this helps!