Elastic benchmark changes

I was looking at the benchmarks for elasticsearch you chaps publish at https://benchmarks.elastic.co/index.html

It seems that around the end of February 2016 there's a big change in the performance characteristics of the tests. Is there a specific change that was made around then? If so what was it?


Hi Tom,

if you hover the graphs at this point in time, it says that we changed the benchmark data set from a log based benchmark to a geonames based benchmark. So this is totally unrelated to changes in Elasticsearch.

Note that the scripts that produce the benchmark results are not available as open source (it is also a bit hard to set them up). That's why we've built Rally. It is available as open source, actively maintained by us and much easier to set up. Also the benchmark data sets are openly available so you can reproduce the benchmark results yourself.

We publish similar benchmark results for the geonames data set produced by Rally at https://elasticsearch-benchmarks.elastic.co/ - Some of the graphs are a bit bumpy before mid of April. This is because Rally is rather new but we still actively look for problems in the benchmark driver and improve it continuously. In summary, I'd say starting with mid of April the data are quite reliable.

We're also about to add new data sets (logging, full text etc.) for Rally but we will not mix different benchmark data sets but rather create new dashboards for them. For more details see also the release blog post for Rally and don't hesitate to ask questions about Rally here in the Elasticsearch forum.