Rally for aggregations on existing ES cluster


I have an existing ElasticSearch Cluster with 36 Data Nodes and 5 Masters.
Our Main Use case is search + aggregations(alot)
It frequency goes to RED state with some concurrent requests.
After reading many blogs, I suspect some of the settings we have done are wrong.

In order to test the performance of ES after tweaking those settings , I am thinking of using rally.
So I have 2 questions:-

  1. Is Rally the right tool for this?
  2. How can I create a test-track which has only search+aggregation queries.
    Please note: I found some articles on search queries but none on Aggregations Queries.

Any help is appreciated.

Running benchmarks against existing clusters with existing indices is in my opinion a very common use-case and I wish there was a proper example of this in the docs. I believe Rally still requires some index specifications even if they are not used, so it is possible to create a slimmed down challenge to do exactly what you are requesting. In order to create complex queries and the distribution of query parameters, you may need to create a custom parameter source in Python. An example of how to create complex multisearch queries with aggregations can be foundin the rally-eventdata-track where it is ued to simulate Kibana queries.

Thanks Christian.
I will look into it.

Yea you rightly said, there are no proper examples of a very common use case - benchmarking existing cluster

Hey @Christian_Dahlqvist !

re: examples of running benchmarks against an existing cluster there is: https://esrally.readthedocs.io/en/stable/recipes.html#benchmarking-an-existing-cluster

Is this sufficient, or is there something missing in your opinion?


What I am looking for is a guide on how to create a custom track that will run custom queries against an existing cluster with existing data. This is something I have not found documented very well.


Hi Liappis

Yes what Christian mentioned is exactly what we are looking for.
It would be very helpful if the same can be closed soon.

I forgot to mention that a very minimal example of a query-only track that doesn't do any index creation can be found in: https://esrally.readthedocs.io/en/stable/track.html#a-track-with-a-single-task.

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.