Thank you for your interest in Rally. I've never really used JMeter or nGrinder, but the benefits of Rally is that it's really tailored towards Elasticsearch.
System setup: Rally knows how to install and start Elasticsearch itself (being for a revision or for a released version) with easy ways to configure settings such as the heap.
Warm up: You can tell Rally to ignore initial iterations to avoid skewing results.
Workload: Rally offers many tracks modeling various use cases with useful initial settings. It uses an official Elasticsearch client, which is more realistic than just sending raw HTTP requests. It allows using advanced features from the API easily. You can also create your own tracks, modeling your data and workload more accurately. You can tune the number of clients or the size of bulk index calls easily. It measures latency/service time/throughput very carefully.
Bugs: We use Rally extensively at Elastic and with the interface Rally provides it's more difficult to run into bugs, while making mistakes using a generic tool is easier
Accidental bottlenecks: Despite being written in Python, Rally is carefully optimized where it matters. For example, in various places we do not parse the JSON output of Elasticsearch.
Use a Structured process: Using the Rally commands to install/run Elasticsearch and having your workload stored in version control makes it much easier to have a repeatable process.
Statistics: Rally provides many metrics related to your benchmarks. Some of them are reported at the end of each run. You can also get additional insights using telemetry devices.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.