How to increase the number of shards or replicas dynamically in single race on esrally

I planned to set cluster and increase the number of shards or replicas to optimize it testing the cluster with esrally

Is there any way to start one shard and increase to n shards on single esrally race

+) I found following graph here (Benchmarking and sizing your Elasticsearch cluster for logs and metrics | Elastic Blog)

is this conducted by increasing the number of nodes one by one during single esrally race? or by testing and summarize tests (conduct one-node test, two-node test and 3-node test separately and put one graph)

Howdy, thanks for your interest in Rally!

Is there any way to start one shard and increase to n shards on single esrally race

You could definitely build a custom track that does this by specifying a series of Operations that creates your target indices/datastreams with incrementing shard counts (see Creating a track from scratch).

However, I'd suggest that it's probably easier to define a single benchmark that instead makes uses of Track Parameters to set the number of primary replica shards. That way you can get seperate 'Race' results for each shard count, e.g.:

$ esrally race [...] --track-params="number_of_shards:1,number_of_replicas:1"
    ____        ____
   / __ \____ _/ / /_  __
  / /_/ / __ `/ / / / / /
 / _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
                /____/

[INFO] Race id is [adaa859b-98e0-4267-ad1c-a9b0983071e8]
[...]

$ esrally race [...] --track-params="number_of_shards:2,number_of_replicas:1"
    ____        ____
   / __ \____ _/ / /_  __
  / /_/ / __ `/ / / / / /
 / _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
                /____/

[INFO] Race id is [fb387eee-729e-4895-a57b-f5a2fb96b6f1]
[...]

The added benefit of having seperate Races is that you can quickly compare their results via the compare subcommand:

$ esrally compare --baseline  adaa859b-98e0-4267-ad1c-a9b0983071e8 --contender fb387eee-729e-4895-a57b-f5a2fb96b6f1

    ____        ____
   / __ \____ _/ / /_  __
  / /_/ / __ `/ / / / / /
 / _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
                /____/


Comparing baseline
  Race ID: adaa859b-98e0-4267-ad1c-a9b0983071e8 
  Race timestamp: 2022-11-23 02:17:25
  Car: external

with contender
  Race ID: fb387eee-729e-4895-a57b-f5a2fb96b6f1
  Race timestamp: 2022-11-23 02:17:14
  Car: external

------------------------------------------------------
    _______             __   _____
   / ____(_)___  ____ _/ /  / ___/_________  ________
  / /_  / / __ \/ __ `/ /   \__ \/ ___/ __ \/ ___/ _ \
 / __/ / / / / / /_/ / /   ___/ / /__/ /_/ / /  /  __/
/_/   /_/_/ /_/\__,_/_/   /____/\___/\____/_/   \___/
------------------------------------------------------
            
|                                                        Metric |   Task |        Baseline |      Contender |          Diff |   Unit |   Diff % |
|--------------------------------------------------------------:|-------:|----------------:|---------------:|--------------:|-------:|---------:|
|                    Cumulative indexing time of primary shards |        |      0.1413     |    0.0124833   |      -0.12882 |    min |  -91.17% |
|             Min cumulative indexing time across primary shard |        |      0.02305    |    0.000383333 |      -0.02267 |    min |  -98.34% |
|          Median cumulative indexing time across primary shard |        |      0.0243667  |    0.00188333  |      -0.02248 |    min |  -92.27% |
|             Max cumulative indexing time across primary shard |        |      0.0455     |    0.00758333  |      -0.03792 |    min |  -83.33% |
|           Cumulative indexing throttle time of primary shards |        |      0          |    0           |       0       |    min |    0.00% |
|    Min cumulative indexing throttle time across primary shard |        |      0          |    0           |       0       |    min |    0.00% |
| Median cumulative indexing throttle time across primary shard |        |      0          |    0           |       0       |    min |    0.00% |
|    Max cumulative indexing throttle time across primary shard |        |      0          |    0           |       0       |    min |    0.00% |
|                       Cumulative merge time of primary shards |        |      0          |    0           |       0       |    min |    0.00% |
|                      Cumulative merge count of primary shards |        |      0          |    0           |       0       |        |    0.00% |
|                Min cumulative merge time across primary shard |        |      0          |    0           |       0       |    min |    0.00% |
|             Median cumulative merge time across primary shard |        |      0          |    0           |       0       |    min |    0.00% |
|                Max cumulative merge time across primary shard |        |      0          |    0           |       0       |    min |    0.00% |
|              Cumulative merge throttle time of primary shards |        |      0          |    0           |       0       |    min |    0.00% |
|       Min cumulative merge throttle time across primary shard |        |      0          |    0           |       0       |    min |    0.00% |
|    Median cumulative merge throttle time across primary shard |        |      0          |    0           |       0       |    min |    0.00% |
|       Max cumulative merge throttle time across primary shard |        |      0          |    0           |       0       |    min |    0.00% |
|                     Cumulative refresh time of primary shards |        |      0.0325833  |    0.00103333  |      -0.03155 |    min |  -96.83% |
|                    Cumulative refresh count of primary shards |        |     16          |   13           |      -3       |        |  -18.75% |
|              Min cumulative refresh time across primary shard |        |      0.00126667 |    0           |      -0.00127 |    min | -100.00% |
|           Median cumulative refresh time across primary shard |        |      0.00128333 |    0.000216667 |      -0.00107 |    min |  -83.12% |
|              Max cumulative refresh time across primary shard |        |      0.02745    |    0.000466667 |      -0.02698 |    min |  -98.30% |
|                       Cumulative flush time of primary shards |        |      0          |    0           |       0       |    min |    0.00% |
|                      Cumulative flush count of primary shards |        |      0          |    0           |       0       |        |    0.00% |
|                Min cumulative flush time across primary shard |        |      0          |    0           |       0       |    min |    0.00% |
|             Median cumulative flush time across primary shard |        |      0          |    0           |       0       |    min |    0.00% |
|                Max cumulative flush time across primary shard |        |      0          |    0           |       0       |    min |    0.00% |
|                                       Total Young Gen GC time |        |      0.021      |    0           |      -0.021   |      s | -100.00% |
|                                      Total Young Gen GC count |        |      1          |    0           |      -1       |        | -100.00% |
|                                         Total Old Gen GC time |        |      0          |    0           |       0       |      s |    0.00% |
|                                        Total Old Gen GC count |        |      0          |    0           |       0       |        |    0.00% |
|                                                    Store size |        |      0.0131358  |    0.000311377 |      -0.01282 |     GB |  -97.63% |
|                                                 Translog size |        |      0.0136973  |    0.000307606 |      -0.01339 |     GB |  -97.75% |
|                                        Heap used for segments |        |      0          |    0           |       0       |     MB |    0.00% |
|                                      Heap used for doc values |        |      0          |    0           |       0       |     MB |    0.00% |
|                                           Heap used for terms |        |      0          |    0           |       0       |     MB |    0.00% |
|                                           Heap used for norms |        |      0          |    0           |       0       |     MB |    0.00% |
|                                          Heap used for points |        |      0          |    0           |       0       |     MB |    0.00% |
|                                   Heap used for stored fields |        |      0          |    0           |       0       |     MB |    0.00% |
|                                                 Segment count |        |     12          |    4           |      -8       |        |  -66.67% |
|                                   Total Ingest Pipeline count |        |      0          |    0           |       0       |        |    0.00% |
|                                    Total Ingest Pipeline time |        |      0          |    0           |       0       |     ms |    0.00% |
|                                  Total Ingest Pipeline failed |        |      0          |    0           |       0       |        |    0.00% |
|                                                Min Throughput |   bulk | 136251          | 2298.03        | -133953       | docs/s |  -98.31% |
|                                               Mean Throughput |   bulk | 136251          | 2298.03        | -133953       | docs/s |  -98.31% |
|                                             Median Throughput |   bulk | 136251          | 2298.03        | -133953       | docs/s |  -98.31% |
|                                                Max Throughput |   bulk | 136251          | 2298.03        | -133953       | docs/s |  -98.31% |
|                                       50th percentile latency |   bulk |   1130.42       |  276.353       |    -854.064   |     ms |  -75.55% |
|                                      100th percentile latency |   bulk |   1752.18       |  381.124       |   -1371.06    |     ms |  -78.25% |
|                                  50th percentile service time |   bulk |   1130.42       |  276.353       |    -854.064   |     ms |  -75.55% |
|                                 100th percentile service time |   bulk |   1752.18       |  381.124       |   -1371.06    |     ms |  -78.25% |
|                                                    error rate |   bulk |      0          |    0           |       0       |      % |    0.00% |


-------------------------------
[INFO] SUCCESS (took 0 seconds)
-------------------------------

+) I found following graph here (Benchmarking and sizing your Elasticsearch cluster for logs and metrics | Elastic Blog)
is this conducted by increasing the number of nodes one by one during single esrally race? or by testing and summarize tests (conduct one-node test, two-node test and 3-node test separately and put one graph)

The chart is showing you results on the Y-Axis and each race on the X-Axis, i.e. these are seperate races. You can see more about using an External Metrics Store here.

Lastly, I recommend watching this talk titlted: The Seven Deadly Sins of Elasticsearch Benchmarking, it does a great job of covering common pitfalls and gotchas when it comes to properly analysing benchmark results.