# \[ESRally Benchmarks\] throughput of search changes every time ES restart
**URL:** https://discuss.elastic.co/t/esrally-benchmarks-throughput-of-search-changes-every-time-es-restart/174631
**Category:** Elasticsearch
**Tags:** rally
**Created:** [March 30, 2019, 8:19am UTC](https://discuss.elastic.co/t/esrally-benchmarks-throughput-of-search-changes-every-time-es-restart/174631 "2019-03-30T08:19:11Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@kakka](https://discuss.elastic.co/u/kakka)
#### Post date: [March 30, 2019, 8:19am UTC](https://discuss.elastic.co/t/esrally-benchmarks-throughput-of-search-changes-every-time-es-restart/174631/1 "2019-03-30T08:19:11Z")
I am using rally to benchmark ES, and track is geonames. Every time when ES starts, run several match\_all tests to warm up and get best throughput. The throughputs is between 45 and 200 in different test. Here is the environment configuration.
# Rally
- version 1.0.0
## operations
```auto
{
"name": "default",
"operation-type": "search",
"body": {
"query": {
"match_all": {}
}
}
},
```
## challenges
```auto
"schedule": [
{
"operation": "default",
"clients": 4,
"warmup-iterations": 500,
"iterations": 1000,
"target-throughput": 210
}
]
```
# Hardware & OS
- Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
- Memory 100G every numa
- SSD
- Linux 3.10.0
- ES binds **4** cores using numactl
# JVM
- -Xms8g
- -Xmx8g
- -XX:NewRatio=2
- -XX:+UseConcMarkSweepGC
# ES
- version 6.2.3
```auto
$ curl -XGET 'http://localhost:9200/_cluster/stats?pretty'
{
"_nodes" : {
"total" : 1,
"successful" : 1,
"failed" : 0
},
"cluster_name" : "elasticsearch",
"timestamp" : 1553929322953,
"status" : "green",
"indices" : {
"count" : 1,
"shards" : {
"total" : 5,
"primaries" : 5,
"replication" : 0.0,
"index" : {
"shards" : {
"min" : 5,
"max" : 5,
"avg" : 5.0
},
"primaries" : {
"min" : 5,
"max" : 5,
"avg" : 5.0
},
"replication" : {
"min" : 0.0,
"max" : 0.0,
"avg" : 0.0
}
}
},
"docs" : {
"count" : 10320000,
"deleted" : 0
},
"store" : {
"size_in_bytes" : 2795283680
},
...
```
---
### Author:  [@zqc0512](https://discuss.elastic.co/u/zqc0512)
#### Post date: [April 1, 2019, 5:44am UTC](https://discuss.elastic.co/t/esrally-benchmarks-throughput-of-search-changes-every-time-es-restart/174631/2 "2019-04-01T05:44:07Z")
change the bulk size and try.
---
### Author:  [@kakka](https://discuss.elastic.co/u/kakka)
#### Post date: [April 1, 2019, 7:09am UTC](https://discuss.elastic.co/t/esrally-benchmarks-throughput-of-search-changes-every-time-es-restart/174631/3 "2019-04-01T07:09:26Z")
In this test, anything about bulk size?
---
### Author:  [@dliappis](https://discuss.elastic.co/u/dliappis)
#### Post date: [April 1, 2019, 7:13am UTC](https://discuss.elastic.co/t/esrally-benchmarks-throughput-of-search-changes-every-time-es-restart/174631/4 "2019-04-01T07:13:56Z")
Hello,
I am not sure what the question is here. If it is why the query throughput varies between 45 and 200 and is not stable, I'd initially think that simply the benchmark over stresses (some aspect of) the cluster making it unstable.
When performing throughput benchmarks it is highly recommended to not only check the achieved target throughput (obviously if it never gets achieved it's the first red flag making the benchmark invalid and lower throughput rates needs to be used) but equally importantly, the [service time](https://esrally.readthedocs.io/en/stable/summary_report.html?highlight=latency#service-time) and [latency](https://esrally.readthedocs.io/en/stable/summary_report.html?highlight=latency#latency). If latency starts growing it indicates that Elasticsearch is slower to service (service\_time) your requests than what would be required to satisfy your target throughput.
Rgs,
Dimitris
---
### Author:  [@kakka](https://discuss.elastic.co/u/kakka)
#### Post date: [April 1, 2019, 8:54am UTC](https://discuss.elastic.co/t/esrally-benchmarks-throughput-of-search-changes-every-time-es-restart/174631/6 "2019-04-01T08:54:56Z")
When throughput is 40, adjust throughput target to 60 and throughput do not grow up.
Here is flame graphs for different throuphput.
- throughput: 44
- throughput: 197
---
### Author:  [@kakka](https://discuss.elastic.co/u/kakka)
#### Post date: [April 26, 2019, 2:35am UTC](https://discuss.elastic.co/t/esrally-benchmarks-throughput-of-search-changes-every-time-es-restart/174631/7 "2019-04-26T02:35:21Z")
Anyboby has idea about this problem?
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [May 24, 2019, 2:35am UTC](https://discuss.elastic.co/t/esrally-benchmarks-throughput-of-search-changes-every-time-es-restart/174631/8 "2019-05-24T02:35:23Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.