# \[search\_phase\_execution\_exception Caused by: es\_rejected\_execution\_exception: rejected execution

**URL:** <https://discuss.elastic.co/t/search-phase-execution-exception-caused-by-es-rejected-execution-exception-rejected-execution/342081>\
**Category:** Elasticsearch\
**Created:** [August 31, 2023, 6:37pm UTC](https://discuss.elastic.co/t/search-phase-execution-exception-caused-by-es-rejected-execution-exception-rejected-execution/342081 "2023-08-31T18:37:58Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![agonzalez](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@agonzalez](https://discuss.elastic.co/u/agonzalez)\
**Post date:** [August 31, 2023, 6:37pm UTC](https://discuss.elastic.co/t/search-phase-execution-exception-caused-by-es-rejected-execution-exception-rejected-execution/342081/1 "2023-08-31T18:37:58Z")

</div>

I am using grafana k6 stress tool to generate some load in elastic and sooner or later i always get this error es\_rejected\_execution\_exception. All are search operations to get 30 docs.

- On small indices of 20-30mb it works quite fine and respone time is 1sec until it gets to 500 virtual users and give the error.

- On other indices that are 300mb it fails quite early just after a few requests and response time get to 14s and fails.

Same happen if we just do 3000 sequential requests that is not much instead of parallel users, i think this is very low load for elastic. After it get saturated elastic start rejecting queries and show this error in monitoris: Node xxx is reporting 10800 search rejections at day XXX

Elastic is 8.9.1 on k8s 1 node with 8GB and 4GB heap size

In monitoring i dont see any metric going very high.

Anyone can explain wthat is this error and why is caused? memory? cpu? any optimization to do?

Error:

> [search\_phase\_execution\_exception Caused by: es\_rejected\_execution\_exception: rejected execution of TimedRunnable{original=org.elasticsearch.action.search.FetchSearchPhase$1@115599a3, creationTimeNanos=36855012795314, startTimeNanos=0, finishTimeNanos=-1, failedOrRejected=false} on TaskExecutionTimeTrackingEsThreadPoolExecutor[name = elasticsearch-es-master-0/search, queue capacity = 1000, task execution EWMA = 164ms, total task execution time = 18m, org.elasticsearch.common.util.concurrent.TaskExecutionTimeTrackingEsThreadPoolExecutor@20706e70[Running, pool size = 4, active threads = 4, queued tasks = 1000, completed tasks = 144548]] Root causes: es\_rejected\_execution\_exception: rejected execution of TimedRunnable{original=ActionRunnable#wrap[org.elasticsearch.search.SearchService$$Lambda$7936/0x00007f344d306668@33107600], creationTimeNanos=36855012420891, startTimeNanos=0, finishTimeNanos=-1, failedOrRejected=false} on TaskExecutionTimeTrackingEsThreadPoolExecutor[name = elasticsearch-es-master-0/search, queue capacity = 1000, task execution EWMA = 164ms, total task execution time = 18m, org.elasticsearch.common.util.concurrent.TaskExecutionTimeTrackingEsThreadPoolExecutor@20706e70[Running, pool size = 4, active threads = 4, queued tasks = 1000, completed tasks = 144548]]: search\_phase\_execution\_exception Caused by: es\_rejected\_execution\_exception: rejected execution of TimedRunnable{original=org.elasticsearch.action.search.FetchSearchPhase$1@115599a3, creationTimeNanos=36855012795314, startTimeNanos=0, finishTimeNanos=-1, failedOrRejected=false} on TaskExecutionTimeTrackingEsThreadPoolExecutor[name = elasticsearch-es-master-0/search, queue capacity = 1000, task execution EWMA = 164ms, total task execution time = 18m, org.elasticsearch.common.util.concurrent.TaskExecutionTimeTrackingEsThreadPoolExecutor@20706e70[Running, pool size = 4, active threads = 4, queued tasks = 1000, completed tasks = 144548]] Root causes: es\_rejected\_execution\_exception: rejected execution of TimedRunnable{original=ActionRunnable#wrap[org.elasticsearch.search.SearchService$$Lambda$7936/0x00007f344d306668@33107600], creationTimeNanos=36855012420891, startTimeNanos=0, finishTimeNanos=-1, failedOrRejected=false} on TaskExecutionTimeTrackingEsThreadPoolExecutor[name = elasticsearch-es-master-0/search, queue capacity = 1000, task execution EWMA = 164ms, total task execution time = 18m, org.elasticsearch.common.util.concurrent.TaskExecutionTimeTrackingEsThreadPoolExecutor@20706e70[Running, pool size = 4, active threads = 4, queued tasks = 1000, completed tasks = 144548]]]:

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [August 31, 2023, 6:45pm UTC](https://discuss.elastic.co/t/search-phase-execution-exception-caused-by-es-rejected-execution-exception-rejected-execution/342081/2 "2023-08-31T18:45:56Z")

</div>

What is the full output of the [cluster stats API](https://www.elastic.co/guide/en/elasticsearch/reference/8.9/cluster-stats.html)?

What is the CPU allocation?

---

<div class="post-metadata">

**Author:** ![agonzalez](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@agonzalez](https://discuss.elastic.co/u/agonzalez)\
**Post date:** [August 31, 2023, 8:34pm UTC](https://discuss.elastic.co/t/search-phase-execution-exception-caused-by-es-rejected-execution-exception-rejected-execution/342081/3 "2023-08-31T20:34:49Z")

</div>

This is output of /\_cluster/stats while doing the testing queries:

CPU requests and limit it is 2 CPUs

> {  
> "\_nodes": {  
> "total": 1,  
> "successful": 1,  
> "failed": 0  
> },  
> "cluster\_name": "elasticsearch",  
> "cluster\_uuid": "xxx",  
> "timestamp": 1693514057160,  
> "status": "yellow",  
> "indices": {  
> "count": 190,  
> "shards": {  
> "total": 190,  
> "primaries": 190,  
> "replication": 0,  
> "index": {  
> "shards": {  
> "min": 1,  
> "max": 1,  
> "avg": 1  
> },  
> "primaries": {  
> "min": 1,  
> "max": 1,  
> "avg": 1  
> },  
> "replication": {  
> "min": 0,  
> "max": 0,  
> "avg": 0  
> }  
> }  
> },  
> "docs": {  
> "count": 37647365,  
> "deleted": 2764735  
> },  
> "store": {  
> "size\_in\_bytes": 14779611603,  
> "total\_data\_set\_size\_in\_bytes": 14779611603,  
> "reserved\_in\_bytes": 0  
> },  
> "fielddata": {  
> "memory\_size\_in\_bytes": 110056,  
> "evictions": 0,  
> "global\_ordinals": {  
> "build\_time\_in\_millis": 580  
> }  
> },  
> "query\_cache": {  
> "memory\_size\_in\_bytes": 8539942,  
> "total\_count": 1163459,  
> "hit\_count": 422535,  
> "miss\_count": 740924,  
> "cache\_size": 5563,  
> "cache\_count": 13079,  
> "evictions": 7516  
> },  
> "completion": {  
> "size\_in\_bytes": 0  
> },  
> "segments": {  
> "count": 1178,  
> "memory\_in\_bytes": 0,  
> "terms\_memory\_in\_bytes": 0,  
> "stored\_fields\_memory\_in\_bytes": 0,  
> "term\_vectors\_memory\_in\_bytes": 0,  
> "norms\_memory\_in\_bytes": 0,  
> "points\_memory\_in\_bytes": 0,  
> "doc\_values\_memory\_in\_bytes": 0,  
> "index\_writer\_memory\_in\_bytes": 317696,  
> "version\_map\_memory\_in\_bytes": 179,  
> "fixed\_bit\_set\_memory\_in\_bytes": 961688,  
> "max\_unsafe\_auto\_id\_timestamp": 1693502487357,  
> "file\_sizes": {}  
> },  
> "mappings": {  
> "total\_field\_count": 16758,  
> "total\_deduplicated\_field\_count": 7840,  
> "total\_deduplicated\_mapping\_size\_in\_bytes": 76399,  
> "field\_types": [  
> {  
> "name": "alias",  
> "count": 76,  
> "index\_count": 1,  
> "script\_count": 0  
> },  
> {  
> "name": "binary",  
> "count": 2,  
> "index\_count": 2,  
> "script\_count": 0  
> },  
> {  
> "name": "boolean",  
> "count": 327,  
> "index\_count": 69,  
> "script\_count": 0  
> },  
> {  
> "name": "constant\_keyword",  
> "count": 6,  
> "index\_count": 2,  
> "script\_count": 0  
> },  
> {  
> "name": "date",  
> "count": 779,  
> "index\_count": 168,  
> "script\_count": 0  
> },  
> {  
> "name": "date\_range",  
> "count": 6,  
> "index\_count": 6,  
> "script\_count": 0  
> },  
> {  
> "name": "double",  
> "count": 93,  
> "index\_count": 19,  
> "script\_count": 0  
> },  
> {  
> "name": "flattened",  
> "count": 39,  
> "index\_count": 6,  
> "script\_count": 0  
> },  
> {  
> "name": "float",  
> "count": 168,  
> "index\_count": 43,  
> "script\_count": 0  
> },  
> {  
> "name": "geo\_point",  
> "count": 24,  
> "index\_count": 3,  
> "script\_count": 0  
> },  
> {  
> "name": "half\_float",  
> "count": 56,  
> "index\_count": 14,  
> "script\_count": 0  
> },  
> {  
> "name": "integer",  
> "count": 648,  
> "index\_count": 101,  
> "script\_count": 0  
> },  
> {  
> "name": "ip",  
> "count": 53,  
> "index\_count": 5,  
> "script\_count": 0  
> },  
> {  
> "name": "keyword",  
> "count": 7235,  
> "index\_count": 173,  
> "script\_count": 0  
> },  
> {  
> "name": "long",  
> "count": 2221,  
> "index\_count": 107,  
> "script\_count": 0  
> },  
> {  
> "name": "match\_only\_text",  
> "count": 270,  
> "index\_count": 3,  
> "script\_count": 0  
> },  
> {  
> "name": "nested",  
> "count": 385,  
> "index\_count": 80,  
> "script\_count": 0  
> },  
> {  
> "name": "object",  
> "count": 2153,  
> "index\_count": 51,  
> "script\_count": 0  
> },  
> {  
> "name": "scaled\_float",  
> "count": 24,  
> "index\_count": 6,  
> "script\_count": 0  
> },  
> {  
> "name": "short",  
> "count": 89,  
> "index\_count": 44,  
> "script\_count": 0  
> },  
> {  
> "name": "text",  
> "count": 1991,  
> "index\_count": 157,  
> "script\_count": 0  
> },  
> {  
> "name": "version",  
> "count": 13,  
> "index\_count": 13,  
> "script\_count": 0  
> },  
> {  
> "name": "wildcard",  
> "count": 100,  
> "index\_count": 37,  
> "script\_count": 0  
> }  
> ],  
> "runtime\_field\_types":   
> },  
> "analysis": {  
> "char\_filter\_types": ,  
> "tokenizer\_types": [  
> {  
> "name": "path\_hierarchy",  
> "count": 124,  
> "index\_count": 124  
> }  
> ],  
> "filter\_types": [  
> {  
> "name": "lowercase",  
> "count": 124,  
> "index\_count": 124  
> }  
> ],  
> "analyzer\_types": [  
> {  
> "name": "custom",  
> "count": 372,  
> "index\_count": 124  
> }  
> ],  
> "built\_in\_char\_filters": ,  
> "built\_in\_tokenizers": [  
> {  
> "name": "standard",  
> "count": 124,  
> "index\_count": 124  
> },  
> {  
> "name": "whitespace",  
> "count": 124,  
> "index\_count": 124  
> }  
> ],  
> "built\_in\_filters": ,  
> "built\_in\_analyzers": [  
> {  
> "name": "ana\_path\_lc",  
> "count": 77,  
> "index\_count": 27  
> },  
> {  
> "name": "ana\_s\_lc",  
> "count": 1353,  
> "index\_count": 123  
> },  
> {  
> "name": "ana\_w\_lc",  
> "count": 1243,  
> "index\_count": 123  
> }  
> ]  
> },  
> "versions": [  
> {  
> "version": "8.4.0",  
> "index\_count": 82,  
> "primary\_shard\_count": 82,  
> "total\_primary\_bytes": 2946839713  
> },  
> {  
> "version": "8.6.2",  
> "index\_count": 71,  
> "primary\_shard\_count": 71,  
> "total\_primary\_bytes": 7233327463  
> },  
> {  
> "version": "8.9.0",  
> "index\_count": 33,  
> "primary\_shard\_count": 33,  
> "total\_primary\_bytes": 4247081827  
> },  
> {  
> "version": "8.9.1",  
> "index\_count": 4,  
> "primary\_shard\_count": 4,  
> "total\_primary\_bytes": 352362600  
> }  
> ],  
> "search": {  
> "total": 94783,  
> "queries": {  
> "regexp": 2,  
> "match\_phrase": 5,  
> "bool": 82800,  
> "terms": 10696,  
> "prefix": 420,  
> "match": 6630,  
> "match\_phrase\_prefix": 1,  
> "exists": 12470,  
> "range": 26036,  
> "term": 91921,  
> "nested": 183,  
> "simple\_query\_string": 387  
> },  
> "sections": {  
> "highlight": 19,  
> "runtime\_mappings": 16,  
> "stored\_fields": 38,  
> "query": 94348,  
> "script\_fields": 38,  
> "pit": 39,  
> "\_source": 16025,  
> "fields": 54,  
> "collapse": 3659,  
> "aggs": 18119  
> }  
> }  
> },  
> "nodes": {  
> "count": {  
> "total": 1,  
> "coordinating\_only": 0,  
> "data": 1,  
> "data\_cold": 0,  
> "data\_content": 0,  
> "data\_frozen": 0,  
> "data\_hot": 0,  
> "data\_warm": 0,  
> "index": 0,  
> "ingest": 1,  
> "master": 1,  
> "ml": 0,  
> "remote\_cluster\_client": 1,  
> "search": 0,  
> "transform": 0,  
> "voting\_only": 0  
> },  
> "versions": [  
> "8.9.1"  
> ],  
> "os": {  
> "available\_processors": 2,  
> "allocated\_processors": 2,  
> "names": [  
> {  
> "name": "Linux",  
> "count": 1  
> }  
> ],  
> "pretty\_names": [  
> {  
> "pretty\_name": "Ubuntu 20.04.6 LTS",  
> "count": 1  
> }  
> ],  
> "architectures": [  
> {  
> "arch": "amd64",  
> "count": 1  
> }  
> ],  
> "mem": {  
> "total\_in\_bytes": 8589934592,  
> "adjusted\_total\_in\_bytes": 8589934592,  
> "free\_in\_bytes": 2310959104,  
> "used\_in\_bytes": 6278975488,  
> "free\_percent": 27,  
> "used\_percent": 73  
> }  
> },  
> "process": {  
> "cpu": {  
> "percent": 100  
> },  
> "open\_file\_descriptors": {  
> "min": 2062,  
> "max": 2062,  
> "avg": 2062  
> }  
> },  
> "jvm": {  
> "max\_uptime\_in\_millis": 11587715,  
> "versions": [  
> {  
> "version": "20.0.2",  
> "vm\_name": "OpenJDK 64-Bit Server VM",  
> "vm\_version": "20.0.2+9-78",  
> "vm\_vendor": "Oracle Corporation",  
> "bundled\_jdk": true,  
> "using\_bundled\_jdk": true,  
> "count": 1  
> }  
> ],  
> "mem": {  
> "heap\_used\_in\_bytes": 1405540880,  
> "heap\_max\_in\_bytes": 4294967296  
> },  
> "threads": 58  
> },  
> "fs": {  
> "total\_in\_bytes": 20957446144,  
> "free\_in\_bytes": 6121500672,  
> "available\_in\_bytes": 6104723456  
> },  
> "plugins": ,  
> "network\_types": {  
> "transport\_types": {  
> "security4": 1  
> },  
> "http\_types": {  
> "security4": 1  
> }  
> },  
> "discovery\_types": {  
> "multi-node": 1  
> },  
> "packaging\_types": [  
> {  
> "flavor": "default",  
> "type": "docker",  
> "count": 1  
> }  
> ],  
> "ingest": {  
> "number\_of\_pipelines": 11,  
> "processor\_stats": {  
> "date": {  
> "count": 0,  
> "failed": 0,  
> "current": 0,  
> "time\_in\_millis": 0  
> },  
> "dot\_expander": {  
> "count": 0,  
> "failed": 0,  
> "current": 0,  
> "time\_in\_millis": 0  
> },  
> "foreach": {  
> "count": 0,  
> "failed": 0,  
> "current": 0,  
> "time\_in\_millis": 0  
> },  
> "geoip": {  
> "count": 0,  
> "failed": 0,  
> "current": 0,  
> "time\_in\_millis": 0  
> },  
> "json": {  
> "count": 0,  
> "failed": 0,  
> "current": 0,  
> "time\_in\_millis": 0  
> },  
> "pipeline": {  
> "count": 0,  
> "failed": 0,  
> "current": 0,  
> "time\_in\_millis": 0  
> },  
> "remove": {  
> "count": 0,  
> "failed": 0,  
> "current": 0,  
> "time\_in\_millis": 0  
> },  
> "rename": {  
> "count": 0,  
> "failed": 0,  
> "current": 0,  
> "time\_in\_millis": 0  
> },  
> "script": {  
> "count": 0,  
> "failed": 0,  
> "current": 0,  
> "time\_in\_millis": 0  
> },  
> "set": {  
> "count": 0,  
> "failed": 0,  
> "current": 0,  
> "time\_in\_millis": 0  
> },  
> "set\_security\_user": {  
> "count": 0,  
> "failed": 0,  
> "current": 0,  
> "time\_in\_millis": 0  
> },  
> "uri\_parts": {  
> "count": 0,  
> "failed": 0,  
> "current": 0,  
> "time\_in\_millis": 0  
> },  
> "user\_agent": {  
> "count": 0,  
> "failed": 0,  
> "current": 0,  
> "time\_in\_millis": 0  
> }  
> }  
> },  
> "indexing\_pressure": {  
> "memory": {  
> "current": {  
> "combined\_coordinating\_and\_primary\_in\_bytes": 0,  
> "coordinating\_in\_bytes": 0,  
> "primary\_in\_bytes": 0,  
> "replica\_in\_bytes": 0,  
> "all\_in\_bytes": 0  
> },  
> "total": {  
> "combined\_coordinating\_and\_primary\_in\_bytes": 0,  
> "coordinating\_in\_bytes": 0,  
> "primary\_in\_bytes": 0,  
> "replica\_in\_bytes": 0,  
> "all\_in\_bytes": 0,  
> "coordinating\_rejections": 0,  
> "primary\_rejections": 0,  
> "replica\_rejections": 0  
> },  
> "limit\_in\_bytes": 0  
> }  
> }  
> },  
> "snapshots": {  
> "current\_counts": {  
> "snapshots": 0,  
> "shard\_snapshots": 0,  
> "snapshot\_deletions": 0,  
> "concurrent\_operations": 0,  
> "cleanups": 0  
> },  
> "repositories": {}  
> }  
> }

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 1, 2023, 4:51am UTC](https://discuss.elastic.co/t/search-phase-execution-exception-caused-by-es-rejected-execution-exception-rejected-execution/342081/4 "2023-09-01T04:51:30Z")

</div>

Even though the index sizes you mentioned are very small, the total size of the data set is around 14GB, which is more than can fit in the operating system page cache. Depending on how you query your data, it is therefore possible that disk I/O could be a bottleneck.

When you run your benchmark, how many indices are you querying in each request? How many indices are queried in total across all requests?

> [@agonzalez](#):
>
> CPU requests and limit it is 2 CPUs

That does not sound like a lot for the number of concurrent queries you are sending. Does the behaviour change if you double this allocation?

---

<div class="post-metadata">

**Author:** ![agonzalez](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@agonzalez](https://discuss.elastic.co/u/agonzalez)\
**Post date:** [September 1, 2023, 5:42pm UTC](https://discuss.elastic.co/t/search-phase-execution-exception-caused-by-es-rejected-execution-exception-rejected-execution/342081/5 "2023-09-01T17:42:40Z")

</div>

I increased cpu limits to 4cpu and same result dont see a difference.

I do 2 load tests , the first test against just one index (200-300mb) and the second test just one index query (small index) I dont mix.

what i notice is when i run the 500 user query test over the bigger index that all are the same search query over and over it reports latency issues it is alway the same query should not be cached?

When i do the same test over the smaller index i dont get any latency issues and run fine.

First test: same simple search query over index tasks (200-300mb) - latency spike to 100ms - 78.30% requests fails

> data\_received..................: 8.0 MB 265 kB/s  
> data\_sent......................: 668 kB 22 kB/s  
> http\_req\_blocked...............: avg=313.14ms min=0s med=0s max=1.63s p(90)=1.17s p(95)=1.44s  
> http\_req\_connecting............: avg=66.66ms min=0s med=0s max=1.05s p(90)=184.18ms p(95)=189.13ms  
> http\_req\_duration..............: avg=5.1s min=39.94ms med=281.84ms max=20.83s p(90)=20.37s p(95)=20.65s  
> { expected\_response:true }...: avg=20.06s min=17.54s med=20.34s max=20.83s p(90)=20.71s p(95)=20.74s  
> http\_req\_failed................: 78.30% ✓ 1061 ✗ 294  
> http\_req\_receiving.............: avg=8.91ms min=0s med=0s max=194.04ms p(90)=6.25ms p(95)=81.78ms  
> http\_req\_sending...............: avg=38.28µs min=0s med=0s max=1ms p(90)=0s p(95)=515.7µs  
> http\_req\_tls\_handshaking.......: avg=240.64ms min=0s med=0s max=1.44s p(90)=959.08ms p(95)=1.26s  
> http\_req\_waiting...............: avg=5.1s min=39.94ms med=281.84ms max=20.83s p(90)=20.32s p(95)=20.62s  
> http\_reqs......................: 1355 45.152482/s  
> iteration\_duration.............: avg=6.42s min=1.04s med=1.28s max=22.47s p(90)=22.01s p(95)=22.22s  
> iterations.....................: 1355 45.152482/s  
> vus............................: 500 min=500 max=500  
> vus\_max........................: 500 min=500 max=500

Second test: same simple query over small index (20-30mb). Latency under 0.5ms

> data\_received..................: 75 MB 2.5 MB/s  
> data\_sent......................: 1.1 MB 36 kB/s  
> http\_req\_blocked...............: avg=62.41ms min=0s med=0s max=1.61s p(90)=0s p(95)=503.03ms  
> http\_req\_connecting............: avg=14.85ms min=0s med=0s max=1.04s p(90)=0s p(95)=145.61ms  
> http\_req\_duration..............: avg=1.45s min=96.47ms med=1.49s max=2.01s p(90)=1.72s p(95)=1.81s  
> { expected\_response:true }...: avg=1.46s min=211.94ms med=1.5s max=2.01s p(90)=1.72s p(95)=1.81s  
> http\_req\_failed................: 0.57% ✓ 34 ✗ 5829  
> http\_req\_receiving.............: avg=537.65µs min=0s med=0s max=201.51ms p(90)=380.84µs p(95)=562.59µs  
> http\_req\_sending...............: avg=26.25µs min=0s med=0s max=1.03ms p(90)=0s p(95)=192.3µs  
> http\_req\_tls\_handshaking.......: avg=47.5ms min=0s med=0s max=1.43s p(90)=0s p(95)=345.57ms  
> http\_req\_waiting...............: avg=1.45s min=96.47ms med=1.49s max=2.01s p(90)=1.72s p(95)=1.8s  
> http\_reqs......................: 5863 195.33764/s  
> iteration\_duration.............: avg=2.52s min=1.1s med=2.5s max=4.35s p(90)=2.8s p(95)=3.31s  
> iterations.....................: 5720 190.573307/s  
> vus............................: 500 min=500 max=500  
> vus\_max........................: 500 min=500 max=500

I attach a picture of monitoring during both tests 1 and 2 and you can see while running test 1 latency increase a lot.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/f/3f46341f5b874e9f7c1fc8d1a000b9b73389e5ca.png)

why this performance difference between some indexes?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 2, 2023, 6:46am UTC](https://discuss.elastic.co/t/search-phase-execution-exception-caused-by-es-rejected-execution-exception-rejected-execution/342081/6 "2023-09-02T06:46:07Z")

</div>

It looks to me like you are saturating CPU in both benchmarks, albeit for a shorter period in the second test.

I would recommend running multiple tests and gradually increase the number of concurrent requests. Start low and keep track of latency and failure rate at each throughput rate. This way you will see at what level problems start (failures or too high latencies) and that will tell you what your cluster is able to handle. I would also recommend running a realistic query load. Rerunning the same exact query is unlikely to give you useful results.

---

<div class="post-metadata">

**Author:** ![agonzalez](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@agonzalez](https://discuss.elastic.co/u/agonzalez)\
**Post date:** [September 4, 2023, 4:40pm UTC](https://discuss.elastic.co/t/search-phase-execution-exception-caused-by-es-rejected-execution-exception-rejected-execution/342081/7 "2023-09-04T16:40:32Z")

</div>

just using 10 virtual users same results but now not al CPU of 4 cpus is consumed just 50% so it is not CPU bottleneck but latency for index 1 requests goes up a lot to 60ms while same requests to other index no problem.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/e/2e84044f0605c383d8d97e6770d6612f7e5e2728.png)

---

<div class="post-metadata">

**Author:** ![agonzalez](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@agonzalez](https://discuss.elastic.co/u/agonzalez)\
**Post date:** [September 5, 2023, 10:47am UTC](https://discuss.elastic.co/t/search-phase-execution-exception-caused-by-es-rejected-execution-exception-rejected-execution/342081/8 "2023-09-05T10:47:08Z")

</div>

Also why is heap size moving constantly from 3gb to 1gb to 3gb to 1gb and so on? any help on how to check what is causing the latency?

---

<div class="post-metadata">

**Author:** ![agonzalez](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@agonzalez](https://discuss.elastic.co/u/agonzalez)\
**Post date:** [September 7, 2023, 2:07pm UTC](https://discuss.elastic.co/t/search-phase-execution-exception-caused-by-es-rejected-execution-exception-rejected-execution/342081/9 "2023-09-07T14:07:59Z")

</div>

anyone can help understand this latency issue? the disk is the same for all data.

---

<div class="post-metadata">

**Author:** ![agonzalez](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@agonzalez](https://discuss.elastic.co/u/agonzalez)\
**Post date:** [September 14, 2023, 4:39pm UTC](https://discuss.elastic.co/t/search-phase-execution-exception-caused-by-es-rejected-execution-exception-rejected-execution/342081/10 "2023-09-14T16:39:55Z")

</div>

i have found that this only happens when queries are done by NEST client but if the request are done by http requests no problem.

You can see difference between stress loading with http requests or NEST client

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/9/592d86d10949b5a8f89566cd97ecfcb6e2103a7b.png)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 12, 2023, 4:40pm UTC](https://discuss.elastic.co/t/search-phase-execution-exception-caused-by-es-rejected-execution-exception-rejected-execution/342081/11 "2023-10-12T16:40:25Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.