Rally for aggregations on existing ES cluster

Elastic Stack Elasticsearch
1

Hi

I have an existing ElasticSearch Cluster with 36 Data Nodes and 5 Masters.
Our Main Use case is search + aggregations(alot)
It frequency goes to RED state with some concurrent requests.
After reading many blogs, I suspect some of the settings we have done are wrong.

In order to test the performance of ES after tweaking those settings , I am thinking of using rally.
So I have 2 questions:-

  1. Is Rally the right tool for this?
  2. How can I create a test-track which has only search+aggregation queries.
    Please note: I found some articles on search queries but none on Aggregations Queries.

Any help is appreciated.

2

Running benchmarks against existing clusters with existing indices is in my opinion a very common use-case and I wish there was a proper example of this in the docs. I believe Rally still requires some index specifications even if they are not used, so it is possible to create a slimmed down challenge to do exactly what you are requesting. In order to create complex queries and the distribution of query parameters, you may need to create a custom parameter source in Python. An example of how to create complex multisearch queries with aggregations can be foundin the rally-eventdata-track where it is ued to simulate Kibana queries.

3

Thanks Christian.
I will look into it.

Yea you rightly said, there are no proper examples of a very common use case - benchmarking existing cluster

4

Hey @Christian_Dahlqvist !

re: examples of running benchmarks against an existing cluster there is: https://esrally.readthedocs.io/en/stable/recipes.html#benchmarking-an-existing-cluster

Is this sufficient, or is there something missing in your opinion?

Dimitris

5

What I am looking for is a guide on how to create a custom track that will run custom queries against an existing cluster with existing data. This is something I have not found documented very well.

6

Hi Liappis

Yes what Christian mentioned is exactly what we are looking for.
It would be very helpful if the same can be closed soon.

7

I forgot to mention that a very minimal example of a query-only track that doesn't do any index creation can be found in: https://esrally.readthedocs.io/en/stable/track.html#a-track-with-a-single-task.