Scalability issue - Rally benchmark on ES 7.0.1

Hey @cks,

Thanks for your interest in Rally and Elasticsearch. I took the liberty of editing your comment for readability, mostly enclosing data sections using the </> icon.

A similar case has been discussed in this discuss topics and the troubleshooting workflow mentioned there would benefit you a lot, in my opinion; additionally this comment may be useful: Elasticsearch official benchmarking results.

To summarize, there seems to be a bottleneck somewhere.
Have you checked if Rally or the load driving machine gets saturated with your settings (175000 bulk_size and 16 or more indexing clients)? I'd especially look at network utilization followed by cpu and disk.

The throttling indexing message is interesting, do you remember in which scenario this happened? (i.e. single/2 nodes, # of indexing clients, and bulk_size)? Also which Java version are you using?

At some point it's likely you'll hit a limit using a single load driver. For example a 1Gbps link gets easily saturated with >1 nodes. This recipe should help you setup a distributed load test driver when you hit this point.

Dimitris