Relative index time metrics?

[alsyia]

Hello,

I'm running ESRally against an existing (test/staging) cluster (whose traffic I can stop if needed for the benchmark duration) and I would like to compare the resulting benchmark against a new, empty cluster. I'm using --pipeline=benchmark-only in both cases, but the Index time metrics (cumulative indexing time, etc) are not comparable.

My understanding is that this is because the old cluster has been running for quite some time now, and ESRally fetches the Index time metrics only once, at the end of the benchmark, for all indices, including those which have not been modified by the benchmark. Is this correct?

Is there a way to configure Rally so that it either:

• Displays only the difference between start and end values for all metrics
• Collect metrics only for the indices the track uses ?

That would help me getting comparable results, I think.

If I request the index stats API manually on each cluster, for the index the track works on, I get figures that looks comparable to me. Can you confirm this is the method I should use if Rally doesn't support it natively?

Thank you very much!

[RickBoyd]

Hi! What you're asking for is not possible at this time, though we always welcome contributions to Rally if you would find this functionality useful.

Please note that though that even outside of the metrics question, the situation you're describing indicates you should run a cluster restart prior to test execution (see #6 in the blog Seven Tips for Better Elasticsearch Benchmarks)

Hope that helps
Rick B

[alsyia]

Hello,

Thanks for the quick reply! I hadn't realized clusters needed to be restarted before each benchmark (although Rally results have been remarkably consistent in the tests I've done). Thanks for the tip!