What's the meaning of rally operations when starting a track

[marone (Marwane Chahoud)]

Hello,

It's been several times working with rally and every time I feel like it's hard to understand and use. I started an http_logs track against a cluster, and can't find the meaning of the following operations that rally did and couldn't find it in the official documentation:

Running delete-index                                                           [100% done]
Running create-index                                                           [100% done]
Running check-cluster-health                                                   [100% done]
Running index-append                                                           [100% done]
Running refresh-after-index                                                    [100% done]
Running force-merge                                                            [100% done]
Running refresh-after-force-merge                                              [100% done]
Running wait-until-merges-finish                                               [100% done]
Running default                                                                [100% done]
Running term                                                                   [100% done]
Running terms_enum                                                             [100% done]
Running range                                                                  [100% done]
Running 200s-in-range                                                          [100% done]
Running 400s-in-range                                                          [100% done]
Running hourly_agg                                                             [100% done]
Running scroll                                                                 [100% done]
Running desc_sort_timestamp                                                    [100% done]
Running asc_sort_timestamp                                                     [100% done]
Running desc_sort_with_after_timestamp                                         [100% done]
Running asc_sort_with_after_timestamp                                          [100% done]
Running desc_sort_timestamp_can_match_shortcut                                 [100% done]
Running desc_sort_timestamp_no_can_match_shortcut                              [100% done]
Running sort_keyword_can_match_shortcut                                        [100% done]
Running sort_keyword_no_can_match_shortcut                                     [100% done]
Running sort_numeric_can_match_shortcut                                        [100% done]
Running sort_numeric_no_can_match_shortcut                                     [100% done]
Running force-merge-1-seg                                                      [100% done]
Running refresh-after-force-merge-1-seg                                        [100% done]
Running wait-until-merges-1-seg-finish                                         [100% done]
Running desc-sort-timestamp-after-force-merge-1-seg                            [100% done]
Running asc-sort-timestamp-after-force-merge-1-seg                             [100% done]
Running desc-sort-with-after-timestamp-after-force-merge-1-seg                 [100% done]
Running asc-sort-with-after-timestamp-after-force-merge-1-seg                  [100% done][INFO] Racing on track [http_logs], challenge [append-no-conflicts] and car ['external'] with version [8.5.3].

Also I have several questions regarding rally:

1. What's the meaning of index-append is it indexing operation? same for 200s-in-range, etc. if there is a documentation describing each meaning it would be appreciated!
2. the default challenge for http_logs is append-no-conflicts, what's the meaning of append-no-conflicts? also, I tried to run many challenges using a Kubernetes manifest with rally as follow but failed:

# ...
    args:
        [
          "race",
          "--track=http_logs",
          "--pipeline=benchmark-only",
          "--challenge=append-no-conflicts,append-no-conflicts-index-only,append-fast-with-conflicts,significant-text",
          "--track-params=bulk_size:5000,bulk_indexing_clients:10",
          "--target-hosts=my_host",
          "--client-options='use_ssl:false,verify_certs:false,basic_auth_user:elastic,basic_auth_password:xxxxx"
        ]
#....

3. when rally finishes a track, it stores data into 3 index:
   a. rally-races*
   b. rally-results*
   c. rally-metrics*

When I make some visualization using the index rally-metrics I get discrete points on kibana (in this viz I have only 3 points of data, but the whole test took around 3 hours, I expected to have many data points to look continuous on a viz):

Also what about the index rally-results*, it contains the same summary table we get when rally is finished right?
Also if there is any recommendation for "how to analyze" the results of the benchmark it would be helpful.

Thank you,

Marwane.

[Evgenia_Badiyanova]

Hello @marone! Thank you for using Rally! I tried to answer all of your questions. Please let me know if I missed anything.

A track (eg. http_logs) could have different challenges. Each challenge has a schedule of operations it executes. To see what operations it will execute you can run --info flag.
For example: esrally info --track=http_logs --challenge=append-no-conflicts.

the default challenge for http_logs is append-no-conflicts , what's the meaning of append-no-conflicts ?

The info command also output the description of the challenge:

================================================
Challenge [append-no-conflicts] (run by default)
================================================

Indexes the whole document corpus using Elasticsearch default settings. We only adjust the number of replicas as we benchmark a single node cluster and Rally will only start the benchmark if the cluster turns green. Document ids are unique so all index operations are append only. After that a couple of queries are run.

If you don't specify any challenge in the info command it will print all the challenges for a specific track with the description and its task schedule.

What's the meaning of index-append is it indexing operation? same for 200s-in-range, etc. if there is a documentation describing each meaning it would be appreciated!

A good place to see what each task will do is in the track definition. For example for http_logs track defines index-append operation in rally-tracks/http_logs/operations/default.json:

      "name": "index-append",
      "operation-type": "bulk",
      "bulk-size": {{bulk_size | default(5000)}},
      "ingest-percentage": {{ingest_percentage | default(100)}},
      "corpora": "http_logs"
    }

To see what operaton-type bulk do you can use Track Reference - Rally 2.10.0 documentation docs.

Same for 200s-in-range. Which you can see is an operation-type search and also see the query it runs.

For metrics rally creates:

rally-metrics-* There should be quite a few records for each metric name and also a lot of other meta information like track, race-id, sample-type and so on. Please take a look at Metrics - Rally 2.10.0 documentation documentation

rally-results-* will contain a few records (but quite a bit less than rally-metrics) for each race-id which will also have a field name for each metric key. Generally it will contain a superset of the summary report you get at the end of a race.

rally-races-* There is one record per race, contains the env information and other information about a particular race.

(in this viz I have only 3 points of data

In your screenshot you are looking at 24 hours period zoomed out, so the chart might look like it only has 3 points. If you zoom in on your graph do you see more points?

Visualizing metrics: When visualizing the data from rally-metrics-* you will want to add extra filters like track and task or operation and sample-type:normal, unless you want to see warmup values as well. And also name of the metric key. Otherwise the graph would try to plot latency and throughput together.

I tried to run many challenges using a Kubernetes manifest with rally as follow but failed.

It looks like you are trying to specify more than one challenge at once. Could you try one of the challenges?

Also considered a good practice to reset you env to extend possible in between races. For example, drop caches, trim disks or restart Elasticsearch when applicable and possible.

I know that's quite a bit of information. Please let me know if you need any clarifications.

Thank you,
Evgenia

[marone (Marwane Chahoud)]

Thank you so much @Evgenia_Badiyanova for the detailed answer, and yes I had to zoom in and filter on data adding more field in the KQL query like name:"latency" and operation :"term" and operation-type :"search" .... But you pointed to the field sample-type, before adding it to the search bar I have this viz:

But when I add sample-type: normal in the KQL search bar, I end up with this:

So my question are:

• to interpret results, is it ok to consider warmup and normal sample-type ?
• in the second viz with sample-type: normal I only have 3 data points, does it mean the term search query run 3 times?
• last question, how many times each operation is run? For example for term query didn't find a parameter saying how much time rally will execute it.

@Evgenia_Badiyanova

It looks like you are trying to specify more than one challenge at once. Could you try one of the challenges?

Yes with only one challenge its working:

      args:
        [
          "race",
          "--track=http_logs",
          "--pipeline=benchmark-only",
          # "--test-mode",
          "--challenge=append-no-conflicts",
          "--track-params=bulk_size:5000,bulk_indexing_clients:10",
          "--target-hosts=https://myhost",
          "--client-options='use_ssl:true,verify_certs:false,basic_auth_user:elastic,basic_auth_password:xxxxx"
        ]

but when trying to list many challenges to run them all at once like: "--challenge=append-no-conflicts','append-no-conflicts-index-only','append-fast-with-conflicts','significant-text'" or pass them as a list (I read somewhere in the doc that challenge accept a list but forget where) "--challenge=['append-no-conflicts','append-no-conflicts-index-only','append-fast-with-conflicts','significant-text']" rally print an error:
esrally.exceptions.InvalidName: Unknown challenge ['append-no-conflicts','append-no-conflicts-index-only','append-fast-with-conflicts','significant-text'] for track [http_logs]

Thanks again,

Marwane.

[Evgenia_Badiyanova]

Hello @marone! Please see my answers below.

filter on data adding more field in the KQL query like name:"latency" and operation :"term" and operation-type :"search" ... . But you pointed to the field sample-type , before adding it to the search bar I have this viz:

As side note: Consider to filter on a challenge and potentially race-id as well, to avoid seeing data for more than one race in case the metric store is used for other races executed at the same time and with the same metric store.

@marone

to interpret results, is it ok to consider warmup and normal sample-type ?

Generally, the answer is no. The reason for warm up is that the peak performance isn't reached right away due to everything in the system needing a warm up, such as JIT compiler and system cache and so on. For more detail answer please take a look at Seven Tips for Better Elasticsearch Benchmarks | Elastic Blog specifically Warm-up section.

• in the second viz with sample-type: normal I only have 3 data points, does it mean the term search query run 3 times?

I am not sure if you are zoomed in enough in your visualization. For sanity check, could you try to check in discover view or change to count of records to see how many records you got?

last question, how many times each operation is run? For example for term query didn't find a parameter saying how much time rally will execute it.

For queries, track schedule specifies the number warm-up and normal iterations. For example for http_logs: See rally-tracks/http_logs/challenges/common/default-schedule.json. For term query in http_logs it's 500 for warm-up and 100 normal (unless you are running runtime-fields challenge):

{
          "name": "term",
          "operation": "term",
          "warmup-iterations": 500,
          "iterations": 100,
          "target-throughput": 50
        }

but when trying to list many challenges to run them all at once like

Is there a reason why you would like to run more than one challenge at a time?
Rally does not support running more than one challenge at a time. This is because a user would generally want to reset the environment to a known stable state before running any next challenges.

Also, Seven Tips for Better Elasticsearch Benchmarks | Elastic Blog is a good starting point for things to remember when doing any benchmarking.

Thank you,
Evgenia

[marone (Marwane Chahoud)]

Thank you so much @Evgenia_Badiyanova for all the details it cleared my mind! As I am more understanding rally now

@Evgenia_Badiyanova

I am not sure if you are zoomed in enough in your visualization. For sanity check, could you try to check in discover view or change to count of records to see how many records you got?

Yes When going to Discover I found exactly 100 data points

and in Lens I had to set the timestamp interval to 100ms as the 100 measurment fall within a ~3sec range time, therefore this is why I was seeing only 3 data points at first place.

@Evgenia_Badiyanova

Is there a reason why you would like to run more than one challenge at a time?

I wanted to run multiple challenges at one-shot to have all results from running one single test but apparently from this discussion and the blog you mentioned, it's not a good approach .

Thanks again,

Marwane.

[Iqra_malick (Work Mail)]

I ran a track "metricbeat" using following command

esrally race --target-hosts=192.168.0.154:9200 --track=metricbeat --pipeline=benchmark-only  --challenge=append-no-conflicts

But no indices named rally-races* , rally-results* , rally-metrics* are created. Can you please tell why?

[Evgenia_Badiyanova]

Hello @Iqra_malick,

Thank you for using Rally! Did you configure the Elasticsearch metric store?

Metrics can be kept in memory (default) or written to a dedicated Elasticsearch cluster (not the cluster where Rally ran its benchmarks). This can be configured in the [reporting] section of ~/.rally/rally.ini.

Thank you,
Evgenia