# Running Rally against AWS Elasticsearch
**URL:** https://discuss.elastic.co/t/running-rally-against-aws-elasticsearch/185059
**Category:** Elasticsearch
**Tags:** rally
**Created:** [June 10, 2019, 9:25pm UTC](https://discuss.elastic.co/t/running-rally-against-aws-elasticsearch/185059 "2019-06-10T21:25:58Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@Gaurav\_Deshmukh](https://discuss.elastic.co/u/Gaurav_Deshmukh)
#### Post date: [June 10, 2019, 9:25pm UTC](https://discuss.elastic.co/t/running-rally-against-aws-elasticsearch/185059/1 "2019-06-10T21:25:58Z")
I created a simple challenge and a operation with custom Parameter Source. The idea behind this implementation is to hit an AWS managed Elasticsearch cluster with the defined challenge which says run the operation for 200 seconds with 15 clients and hit the max-throughput of 100. The operation is just a simple Parameter Source which currently just returns a static match\_all query. Later on the plan with this operation (just FYI) is to randomly change a field in the query and pass that query.
So when I try to hit run the track with this simple "perf-test" challenge, I notice that on AWS console the search operations/min does not match the to ops/sec you get in the final Rally report.
For instance when I am getting 100 as the max throughput in the Rally report, the max search operations per min in the cloudwatch console is around 34.5k. As far as I understood from the Rally manual, throughput in Rally world is the number of operations per second; where the operations is the simple "match\_all" custom Parameter Source I have created which indirectly indicates an operation is one match-all search query. So from this logic I should be expecting only 6000 search operations/min in the AWS console for the 100 throughput in Rally. But I am not able to find out any reason for this discrepancy.
Am I missing anything here?
# Reference:
## AWS ES cluster information
1. Version: 6.5
## Custom Parameter Source
```auto
class SearchRandomUtteranceSource(QueryParamSource):
def params(self):
query_terms = list(self.terms) # copy
#query_terms.append(str(random.randint(1, 1000))) # avoid caching
default_index = "document"
result = {
"body": {
"query": {
"match_all": {
}
}
},
"index": self._params.get("index", default_index),
"type": None,
# This is the old name (before Rally 0.10.0). Remove after some grace period.
"use_request_cache": self._params.get("cache", False),
"cache": self._params.get("cache", False)
}
#print random.choice(query_terms)
return result
```
## Challenge:
```auto
{
"name": "perf-test",
"description":"Runs a perf test with using param source",
"schedule":[
{
"operation": "search_random_utterance",
"clients": 15,
"time-period": 200,
"target-throughput": 100
}
]
}
```
## Operation:
```auto
{
"name": "search_random_utterance",
"operation-type": "search",
"param-source": "search_random_utterance-source"
}
```
---
### Author:  [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)
#### Post date: [June 11, 2019, 4:28pm UTC](https://discuss.elastic.co/t/running-rally-against-aws-elasticsearch/185059/2 "2019-06-11T16:28:14Z")
Hi,
> [@Gaurav\_Deshmukh](#):
>
> For instance when I am getting 100 as the max throughput in the Rally report, the max search operations per min in the cloudwatch console is around 34.5k. As far as I understood from the Rally manual, throughput in Rally world is the number of operations per second
In terms of Rally your understanding is correct. Rally will attempt to issue the query 100 times per second in total or 6000 times per minute. It might be less than that if your cluster cannot achieve that throughput but it will never be more.
Unfortunately, I don't know anything about Cloudwatch and how it works but I guess it would also issue calls against the Elasticsearch stats APIs (e.g. node stats) to get its monitoring data. However, I'd expect those calls to be fairly infrequent, maybe in the order of one call per second. Also, Rally itself issues a few API calls at the beginning of the benchmark e.g. to determine the Elasticsearch version but also this does not get you nowhere near that number.
The [node usage API](https://www.elastic.co/guide/en/elasticsearch/reference/7.1/cluster-nodes-usage.html) might help you understand what type of requests are issued which might lead you towards discovering what is causing that traffic.
Daniel
---
### Author:  [@martinr\_ubi](https://discuss.elastic.co/u/martinr_ubi)
#### Post date: [June 11, 2019, 10:17pm UTC](https://discuss.elastic.co/t/running-rally-against-aws-elasticsearch/185059/3 "2019-06-11T22:17:50Z")
How many shards do you have for the "document" index?
> [@](#):
>
> `SearchRate`: The total number of search requests per minute for all shards on a node. A single call to the `_search` API might return results from many different shards. If five of these shards are on one node, the node would report 5 for this metric, even though the client only made one request.
>
> Relevant node statistics: Average
>
> Relevant cluster statistics: Average, Maximum, Sum
[Source](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains.html)
If you have 5 shards, that would fit with 30K search rate and the rest of the discrepancy could simply be due to statistical/sampling "artefacts": Which statistic for which time range for which period over which dimension...and the peculiarities that can be specific to each metrics like the effective sampling rate, source of truth, intermediate aggregations, frequency of pull/push for the the source of truth, potential sampling distortions, etc.
It's hard to ponder the possibilities for sure since you haven't provided some details like how many ES nodes you have, how many shards were targeted by your query, where were the shards when the queries ran, which dimension over which period over which time range did you check. (How _exactly_ did you obtain this 34.5K figure... from cloudwatch.)
In short, once the shard count multiplier is understood, if it applies in your case, the rest is more easily explained away because it's no longer a multiplication by 5 of what you expected to see.
If that's not it, I would recommend to ask AWS, something can explain such a discrepancy for sure, if it's not the shard count angle, it's something else.
Last but not least if you're surprised about the shard count multiplier effect in this metric, it's not THAT weird... Lots of stuff/metrics are like that in ES like for example when you check the ES query slow log, the queries are logged per shards... So that same "multiplier" applies in that context too.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 9, 2019, 10:17pm UTC](https://discuss.elastic.co/t/running-rally-against-aws-elasticsearch/185059/4 "2019-07-09T22:17:54Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.