# Benchmarck elastic cluster with rally

**URL:** <https://discuss.elastic.co/t/benchmarck-elastic-cluster-with-rally/279068>\
**Category:** Elasticsearch\
**Tags:** rally\
**Created:** [July 19, 2021, 1:26pm UTC](https://discuss.elastic.co/t/benchmarck-elastic-cluster-with-rally/279068 "2021-07-19T13:26:12Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Abdelhalim](https://avatars.discourse-cdn.com/v4/letter/a/838e76/32.png) [@Abdelhalim](https://discuss.elastic.co/u/Abdelhalim)\
**Post date:** [July 19, 2021, 1:26pm UTC](https://discuss.elastic.co/t/benchmarck-elastic-cluster-with-rally/279068/1 "2021-07-19T13:26:12Z")

</div>

Hello,

I am going to setup a Prod cluster, I receive 30GB of logs per day (Need to keep the data 1 month in Hot node and 11 months in warm node )

I did some maths, and I am going to use (3 hot nodes and 4 warm nodes ) [64GB of memory in each node)

I have never used rally before, that's why I would like to ask some questions :  
1- How can I benchmark this clutser to see how many eps I can get with this cluster configuration ? 2- Should I create the cluster first to be able to benchmark it, or `rally` can simulate it ?

Thanks in advance,  
Best regards

---

<div class="post-metadata">

**Author:** ![Bradley\_Deam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bradley_deam/32/52991_2.png) [@Bradley\_Deam](https://discuss.elastic.co/u/Bradley_Deam)\
**Post date:** [July 20, 2021, 11:01pm UTC](https://discuss.elastic.co/t/benchmarck-elastic-cluster-with-rally/279068/2 "2021-07-20T23:01:34Z")

</div>

Hi [@Abdelhalim](https://discuss.elastic.co/u/Abdelhalim),

Have you had a look through the [`rally` Quickstart docs](https://esrally.readthedocs.io/en/stable/quickstart.html)? This runs through installing `rally` and running a basic benchmark.

We also have some documentation related to..

- [Developing your own custom track](https://esrally.readthedocs.io/en/stable/adding_tracks.html)
- [Tips & Tricks around benchmarking existing clusters etc](https://esrally.readthedocs.io/en/stable/recipes.html)

> 2- Should I create the cluster first to be able to benchmark it, or `rally` can simulate it ?

Rally can do either - but in any case you'll first need to have provisioned the underlying infrastructure (i.e. the machines themselves).

See [Setting up a Cluster](https://esrally.readthedocs.io/en/stable/cluster_management.html) on how to use Rally to provision Elasticsearch for you.

This should be enough to get you started!

Thanks,  
Brad

---

<div class="post-metadata">

**Author:** ![Abdelhalim](https://avatars.discourse-cdn.com/v4/letter/a/838e76/32.png) [@Abdelhalim](https://discuss.elastic.co/u/Abdelhalim)\
**Post date:** [July 22, 2021, 1:22pm UTC](https://discuss.elastic.co/t/benchmarck-elastic-cluster-with-rally/279068/3 "2021-07-22T13:22:09Z")

</div>

Thanks @Bradley_Deam , the documentation helped me to understand how `rally` works,  
I am using the command bellow to test the benchmark :

```auto
esrally race --track=http_logs --target-hosts=10.13.81.11:9200,10.13.81.12:9200,10.13.81.13:9200 --pipeline=benchmark-only --client-options='use_ssl:true,verify_cert:true,ca_certs:'ca.crt',client_cert:'beat.crt',client_key:'beat.key',basic_auth_user:'elastic',basic_auth_password:'My_Passwd'' --track-params='number_of_shards:1' --track-params='number_of_replicas:0'

```

I have just 1 final question :

I would like to benchmark the indexing and searching performance in the same time. for the search for example for 5 users with a targeted throughput of 1000 OPS.

Could you please tell me which options to add to my command ?

Best regards

---

<div class="post-metadata">

**Author:** ![Bradley\_Deam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bradley_deam/32/52991_2.png) [@Bradley\_Deam](https://discuss.elastic.co/u/Bradley_Deam)\
**Post date:** [August 1, 2021, 11:53pm UTC](https://discuss.elastic.co/t/benchmarck-elastic-cluster-with-rally/279068/4 "2021-08-01T23:53:04Z")

</div>

Sorry for the delay here, the notification of your reply got lost in my inbox ☹

> I would like to benchmark the indexing and searching performance in the same time. for the search for example for 5 users with a targeted throughput of 1000 OPS.
> 
> Could you please tell me which options to add to my command ?

Sure - this sounds like a great fit for a `parallel` block - [see the docs here](https://esrally.readthedocs.io/en/stable/track.html#running-tasks-in-parallel). The docs also contain an example of concurrent indexing and querying..

> In this scenario, we run indexing and a few queries in parallel with a total of 14 clients:
> 
> ```auto
> 
> schedule": [
> {
> "parallel": {
> "tasks": [
> {
> "operation": "bulk",
> "warmup-time-period": 120,
> "time-period": 3600,
> "clients": 8,
> "target-throughput": 50
> },
> {
> "operation": "default",
> "clients": 2,
> "warmup-iterations": 50,
> "iterations": 100,
> "target-throughput": 50
> },
> {
> "operation": "term",
> "clients": 2,
> "warmup-iterations": 50,
> "iterations": 100,
> "target-throughput": 200
> },
> {
> "operation": "phrase",
> "clients": 2,
> "warmup-iterations": 50,
> "iterations": 100,
> "target-throughput": 200
> }
> ]
> }
> }
> 
> ```

Note that the `target-throughput` specifies the number of requests per second over all clients, so a `target-throughput: 50` with `clients: 2` means each client will aim to perform ~25 op/s.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 29, 2021, 11:53pm UTC](https://discuss.elastic.co/t/benchmarck-elastic-cluster-with-rally/279068/5 "2021-08-29T23:53:10Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.