# The number of clients in search operation

**URL:** <https://discuss.elastic.co/t/the-number-of-clients-in-search-operation/143255>\
**Category:** Elasticsearch\
**Tags:** rally\
**Created:** [August 7, 2018, 7:21am UTC](https://discuss.elastic.co/t/the-number-of-clients-in-search-operation/143255 "2018-08-07T07:21:22Z")\
**Posts on this page:** 1\
**Showing post:** 9

<div class="post-metadata">

**Author:** ![dliappis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dliappis/32/56174_2.png) [@dliappis](https://discuss.elastic.co/u/dliappis)\
**Post date:** [August 9, 2018, 1:06pm UTC](https://discuss.elastic.co/t/the-number-of-clients-in-search-operation/143255/9 "2018-08-09T13:06:33Z")

</div>

Some inline comments below. Note that without seeing the entire Rally report it's not easy to understand the whole picture.  
Esp. for `service_time` vs `latency`, the [FAQ](http://esrally.readthedocs.io/en/stable/faq.html#what-does-latency-and-service-time-mean-and-how-do-they-related-to-the-took-field-that-elasticsearch-returns) mentioned earlier is an essential read to understand the differences.

> 1.When I initially configured target-throughput:20 and clients:1, I have received the following report  
> max-throughput: 18.6 99th percentile latency: 2391 99th percentile service time : 97

`max-thoughtput` not achieving the set target-throughput and the very high value of 99th percentile latency vs 99th percentile service time indicates that the benchmark is unstable and the numbers are not very representative. Despite of this, I have a few additional, more detailed comments:

With `clients:1` you need to remember what I mentioned earlier, i.e. request-response is blocking.

So if your cluster is slow executing the queries, you can't achieve the configured `target-throughput` and this is what happened here, the max throughput achieved was `18.6ops/s < 20` `target-throughput`.  
`Median Throughput` would also be useful to check.

The 99th percentile service time (`service_time` is time taken between issuing request and receiving response) metric tells us that 99% of requests took `<=97ms` to return.  
It's good to keep in mind that interpretation of those numbers is related to the iterations; e.g. if you had `500` iterations, 99th percentile means that `5` requests ended up having `>97ms` service time. It would be useful to see the other percentiles here e.g. 50th to see how slow a larger % of operations were.

For latency it's best to read the great analogy with the Barista and Coffee shop, mentioned by @danielmitterdorfer[here](https://discuss.elastic.co/t/rally-default-latency-performance-degradation/120763/4)  
TL;DR If requests are constantly taking longer to execute than the calculated schedule, this piles up and can lead to high latency values [1].

> 2.When I initially configured target-throughput:100 and clients:10, I have received the following report  
> max-throughput: 99 99th percentile latency: 395 99th percentile service time : 362

Here you have more clients to issue parallel requests despite the request-response delay and you can see that increasing your clients and target-throughput also increased the service time more than 4x to `362ms`.

This shows that, the more queries you send, the slower your cluster is able to service the queries.  
Note that issuing search requests is a lightweight operation for Rally (there isn't much expensive IO in the background like with `bulk` for example).

> From test 2, we can find the cluster can reach 100 throughput; i wonder why 99th percentile latency in test 1 is so large(2391ms) even though the target-throughput is below 100 ?

> In my case, I think the throughput of my cluster is relatively high, but from the test results, i find the throughput is low. So I wonder if the load generated by rally is too low; I want to know the maximum number of clients in the search operation, and the maximum request per second per client.

The number of clients depends on what you want to achieve; e.g. if you are benchmarking a 300ms SLA for your query responses under a load of 100 queries/s, you'll need at least 30 clients. Rally will schedule load among its clients to achieve the target-throughput and not more.

I would also recommend that you enable an Elasticsearch [metrics store](https://esrally.readthedocs.io/en/stable/configuration.html#configuration-options) so that you can have better visibility into each request. For each `search` operation-type, there will be metric records for `latency`, `service_time` and `throughput` to help you understand better what's happening over time than just the summary at the end.

[1] With `clients:1` and `target-throughput:20`, Rally will ensure that requests are only issued every `1/20=50ms`. If a large amount of operations takes `>50ms` to return, this will pile up and influence the `latency_time`.

---

_[View the full topic](https://discuss.elastic.co/t/the-number-of-clients-in-search-operation/143255)._