Are there docs to help interpret the results?

[flavor8]

Obviously the results from Rally are relative and depend on the cluster configuration. That said, it would be useful if there was a reference guide to explain what would be a good or a bad result for each of the scores. Does such a thing exist?

[danielmitterdorfer]

Hi @flavor8,

there is not so much in the official docs about that but I gave a talk around this topic at Javazone last year (slides). They usually record the talks but there was a technical problem with mine. However another video with a very similar version of this talk is available.

I think it is crucial to understand the relationship between service time, latency and throughput and all these terms are explained in the talk (around 35 minutes in the talk; I overhauled slides quite a bit from the video version to make this even clearer).

I plan to improve the documentation on this front anyway but I am also happy to accept PRs.

If something is not clear after you've watched the talk or looked at the slides (hint: there is bonus material at the end of the slides), please just ask questions here or even raise a doc issue in the Rally Github repo.

Daniel

[flavor8]

Thanks @danielmitterdorfer,

I watched your talk -- good presentation, and informative. We'll reapproach our benchmarking with several improvements to our process.

That said I think it would be useful to have documentation that talked about the relationships between the scores in a healthy configuration. Might be a stretch, but perhaps Rally itself could identify "green", "yellow" or "red" findings, just based on the relative differences rather than the whole numbers.

You might also offer a reference benchmark on a known cluster config. E.g. "results from geodata benchmarks with 1 dedicated master of m4.large & 2 data nodes of r4.large on a well configured(*) cluster". That'd help give some confidence about a given configuration.

Nice work. Thanks!

[danielmitterdorfer]

Hi @flavor8,

glad to hear that the talk has helped you.

@flavor8

That said I think it would be useful to have documentation that talked about the relationships between the scores in a healthy configuration. Might be a stretch, but perhaps Rally itself could identify "green", "yellow" or "red" findings, just based on the relative differences rather than the whole numbers.

Interesting idea; I need to ponder that.

@flavor8

You might also offer a reference benchmark on a known cluster config. E.g. "results from geodata benchmarks with 1 dedicated master of m4.large & 2 data nodes of r4.large on a well configured(*) cluster". That'd help give some confidence about a given configuration.

We have something like this in the making but I fear it will take some time.

Daniel