Racing with multiple laps - Metrics Calculation

[Keyur_Doshi]

Hello,

I've two questions on Metrics calculation.

A. Would like to know more about how metrics in Final score are calculated when raced with multiple laps. I was under the assumption that the final score would be the average of the laps score (summation of score in laps / number of laps). I ran the race with 3 laps and when I observe the final score - for some metrics it is average score of the laps score while for some other metrics it is not average. It would be great if you could explain or point me to the document which explains how the final score are calculated when racing with multiple laps. (Please refer file: race01.md and race02.md at Metrics file location)

B. When running the tournament with two races (both with 3 laps but different warmup-time-period). Metrics for Heap used for points is generated twice (with different values). Not sure if I did something wrong or it is expected to be that way. (Please refer file: Tournament.md at Metrics file location).

Also, it would be great if we could save the tournament report in markdown file. This command is not working esrally compare --baseline=XX --contender=XX --report-file=tournament.md

Thank you in advance.

[danielmitterdorfer]

Hi @Keyur_Doshi,

Would like to know more about how metrics in Final score are calculated when raced with multiple laps.

This really depends on the type of metrics. The basic idea is that you can start by looking at the final score and then look at the lap scores to understand run to run variation better (maybe the variations are something that we should start to show in the summary report...).

• For all request metrics (throughput, service time, latency), it is taking all captured samples into consideration.
• For metrics that describe total times, like indexing time, merge time or GC times, Rally will sum all values up. This applies similarly to metric like total written bytes.
• Resource consumption (like "Heap used for ...") and segment counts are the only metrics where we calculate the median (not the average).

I hope that clarifies that point a bit. If you are interested in the complete details, you can look at reporter.py.

Heap used for points is generated twice

Good catch. That's an error in the labeling, the calculation works fine. The first value is "Heap used for points", the second one is "Heap used for stored fields". I've corrected it on master and the fix will be out with the next release.

Also, it would be great if we could save the tournament report in markdown file. This command is not working esrally compare --baseline=XX --contender=XX --report-file=tournament.md

Good point. Can you please open a ticket on Github for that?

Daniel

[Keyur_Doshi]

Hello @danielmitterdorfer, Thank you for the clarification.

This really depends on the type of metrics. The basic idea is that you can start by looking at the final score and then look at the lap scores to understand run to run variation better (maybe the variations are something that we should start to show in the summary report...).

It would be great if we could have the variations in the summary report.

I hope that clarifies that point a bit. If you are interested in the complete details, you can look at reporter.py.

Sure, I'll deep dive into reporter.py and the clarifications on those points helped.

That's an error in the labeling, the calculation works fine

Thank you for that.[quote="danielmitterdorfer, post:2, topic:77706"]
Good point. Can you please open a ticket on Github for that?
[/quote]

Okay, I've opened a ticked for that - https://github.com/elastic/rally/issues/249

Thank you,
Keyur