# Rally: How do the number of requests get calculated in a cluster?

**URL:** <https://discuss.elastic.co/t/rally-how-do-the-number-of-requests-get-calculated-in-a-cluster/338851>\
**Category:** Elasticsearch\
**Tags:** rally\
**Created:** [July 20, 2023, 6:51am UTC](https://discuss.elastic.co/t/rally-how-do-the-number-of-requests-get-calculated-in-a-cluster/338851 "2023-07-20T06:51:39Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![lquenti](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lquenti/32/116424_2.png) [@lquenti](https://discuss.elastic.co/u/lquenti)\
**Post date:** [July 20, 2023, 6:51am UTC](https://discuss.elastic.co/t/rally-how-do-the-number-of-requests-get-calculated-in-a-cluster/338851/1 "2023-07-20T06:51:39Z")

</div>

Hi,

we want to use clustered rally to do a really large scaling test (n=100 ES nodes, ? load distributors) for different corpora and FS settings.

In order to accomplish that, I have a few questions regarding the actual load generation:

1. For the default benchmarks (lets say `nyc_taxis`), do I understand correctly that if in the challenge once the `clients` parameter is not specified that it will be single threaded?

2. Furthermore, if I specify the number of "clients" to 4, and I have two load drivers, do I have 4 or 8 threads sending concurrently?

3. Analagously: If my challenge operation specifies "iterations": 1000, does this mean 1000 iterations in total or 1000 iterations per load driver?

4. Is the worker parallelization multi-processing (no GIL) or multi-threading (GIL)? If the latter, any nice way to specify two daemons running on the same machine without any container isolation?

---

<div class="post-metadata">

**Author:** ![lquenti](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lquenti/32/116424_2.png) [@lquenti](https://discuss.elastic.co/u/lquenti)\
**Post date:** [July 21, 2023, 9:42am UTC](https://discuss.elastic.co/t/rally-how-do-the-number-of-requests-get-calculated-in-a-cluster/338851/2 "2023-07-21T09:42:17Z")

</div>

Also, if the number of "clients" are the number of global clients, what would happen if "clients" \< number of rally daemons available? 😃

---

<div class="post-metadata">

**Author:** ![Bradley\_Deam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bradley_deam/32/52991_2.png) [@Bradley\_Deam](https://discuss.elastic.co/u/Bradley_Deam)\
**Post date:** [July 24, 2023, 12:30am UTC](https://discuss.elastic.co/t/rally-how-do-the-number-of-requests-get-calculated-in-a-cluster/338851/3 "2023-07-24T00:30:27Z")

</div>

Hi @lquenti, thanks for your interest in Rally and the detailed questions. Let me try and answer these in line.

> 1. For the default benchmarks (lets say `nyc_taxis`), do I understand correctly that if in the challenge once the `clients` parameter is not specified that it will be single threaded?

Yes, but I caveat this statement with the fact that each track exposes different parameters, and you can adjust the [concurrency of each task via the `clients` option](https://esrally.readthedocs.io/en/stable/track.html?highlight=clients#schedule). Some tasks do not allow you to adjust the concurrency with track params because there's no template variable to allow so.

> 1. Furthermore, if I specify the number of "clients" to 4, and I have two load drivers, do I have 4 or 8 threads sending concurrently?

From Rally's perspective it allocates clients across all available Workers, and another 'load driver' machine is just more Workers, so you'd only have 4.

> 1. Analagously: If my challenge operation specifies "iterations": 1000, does this mean 1000 iterations in total or 1000 iterations per load driver?

This is the task as a whole, it means 1000 in total.

> 1. Is the worker parallelization multi-processing (no GIL) or multi-threading (GIL)? If the latter, any nice way to specify two daemons running on the same machine without any container isolation?

We use multi-processing. By default the Rally daemon will start a Worker (i.e. a seperate Python process) per-available core on the machine ([see `available.cores` setting to override this](https://esrally.readthedocs.io/en/stable/configuration.html?highlight=available+core#system)), and then starts a single async event loop per Worker.

> Also, if the number of "clients" are the number of global clients, what would happen if "clients" \< number of rally daemons available? 😃

You'd just have Workers without any allocated tasks.

FWIW, pertaining to multi-machine load driver setups, it's uncommon in our experience to actually require this setup unless you're aiming to test _very_ large scale benchmarks with hundreds of thousands, if not millions of requests per-second. Depending on the track and workload, it's entirely feasible for a single 8 core machine to simulate thousands of clients without being the bottleneck.

The 'heavier' the track, the more CPU time you'll need on the load driver. Probably our most [resource intensive track is elastic/logs](https://github.com/elastic/rally-tracks/tree/master/elastic/logs), which dynamically generates documents during indexing, and even with this overhead we've simulated ~1m docs/s on a single load driver (1GB/s network traffic) with 32 cores.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 21, 2023, 12:30am UTC](https://discuss.elastic.co/t/rally-how-do-the-number-of-requests-get-calculated-in-a-cluster/338851/4 "2023-08-21T00:30:32Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.