Using Rally to benchmark a large multi node cluster

[sharathr83 (sharath)]

We has tried Rally (0.7.2) for sometime last year ( Sep 2017) and remember facing a few issues. We had to eventually abandon it on our side because of time constraints we had.

Were there anything major that changed form 0.7.2 to the latest version that would specifically help us benchmark a large multi cluster node ? Is there anyway though which we can do back of the envelope calculations to determine how many load generators we might need to attain the load we want ?

Our aim is to benchmark an existing cluster (we created our own custom tracks)

Thanks !

[danielmitterdorfer]

Hi,

The Rally release page summarizes all changes between releases. We had lots of stability improvements since then and the track format is less verbose.

@sharathr83

Is there anyway though which we can do back of the envelope calculations to determine how many load generators we might need to attain the load we want ?

When you want to know the maximum capacity of a single load generator machine, you could run against a mock target (e.g. nginx returning static responses). Apart from the setup, load generator machines can act independent of each other so they should scale nicely. For benchmarks of larger clusters you need to think about network setup to avoid saturating the network between load generator machines and the Elasticsearch cluster.

Daniel