# SearchPhaseExecutionException caused by EsRejectedExecutionException

**URL:** <https://discuss.elastic.co/t/searchphaseexecutionexception-caused-by-esrejectedexecutionexception/175775>\
**Category:** Elasticsearch\
**Created:** [April 8, 2019, 7:29am UTC](https://discuss.elastic.co/t/searchphaseexecutionexception-caused-by-esrejectedexecutionexception/175775 "2019-04-08T07:29:36Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![igorc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igorc/32/38507_2.png) [@igorc](https://discuss.elastic.co/u/igorc)\
**Post date:** [April 8, 2019, 7:29am UTC](https://discuss.elastic.co/t/searchphaseexecutionexception-caused-by-esrejectedexecutionexception/175775/1 "2019-04-08T07:29:36Z")

</div>

Hi everyone,

I'm getting the following exception in one of the ES data nodes running as stateful-set in kubernetes:

```
[2019-04-08T07:11:59,013][WARN][r.suppressed] [es-data-1] path: /filebeat-*/_search, params: {size=500, index=filebeat-*}
org.elasticsearch.action.search.SearchPhaseExecutionException: 
    at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:293) ~[elasticsearch-6.5.2.jar:6.5.2]
    at org.elasticsearch.action.search.FetchSearchPhase$1.onFailure(FetchSearchPhase.java:91) ~[elasticsearch-6.5.2.jar:6.5.2]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.onRejection(AbstractRunnable.java:63) ~[elasticsearch-6.5.2.jar:6.5.2]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onRejection(ThreadContext.java:713) ~[elasticsearch-6.5.2.jar:6.5.2]
    at org.elasticsearch.common.util.concurrent.TimedRunnable.onRejection(TimedRunnable.java:50) ~[elasticsearch-6.5.2.jar:6.5.2]
    at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.doExecute(EsThreadPoolExecutor.java:104) ~[elasticsearch-6.5.2.jar:6.5.2]
    at org.elasticsearch.common.util.concurrent.QueueResizingEsThreadPoolExecutor.doExecute(QueueResizingEsThreadPoolExecutor.java:86) ~[elasticsearch-6.5.2.jar:6.5.2]
    at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.execute(EsThreadPoolExecutor.java:93) ~[elasticsearch-6.5.2.jar:6.5.2]
    at org.elasticsearch.action.search.AbstractSearchAsyncAction.execute(AbstractSearchAsyncAction.java:308) ~[elasticsearch-6.5.2.jar:6.5.2]
    at org.elasticsearch.action.search.FetchSearchPhase.run(FetchSearchPhase.java:80) ~[elasticsearch-6.5.2.jar:6.5.2]
    at org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:160) ~[elasticsearch-6.5.2.jar:6.5.2]
    at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:153) ~[elasticsearch-6.5.2.jar:6.5.2]
    at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:254) ~[elasticsearch-6.5.2.jar:6.5.2]
    at org.elasticsearch.action.search.InitialSearchPhase.successfulShardExecution(InitialSearchPhase.java:256) ~[elasticsearch-6.5.2.jar:6.5.2]
    at org.elasticsearch.action.search.InitialSearchPhase.onShardResult(InitialSearchPhase.java:244) ~[elasticsearch-6.5.2.jar:6.5.2]
    at org.elasticsearch.action.search.InitialSearchPhase.access$200(InitialSearchPhase.java:48) ~[elasticsearch-6.5.2.jar:6.5.2]
    at org.elasticsearch.action.search.InitialSearchPhase$2.lambda$innerOnResponse$0(InitialSearchPhase.java:217) ~[elasticsearch-6.5.2.jar:6.5.2]
    at org.elasticsearch.action.search.InitialSearchPhase.maybeFork(InitialSearchPhase.java:176) [elasticsearch-6.5.2.jar:6.5.2]
    at org.elasticsearch.action.search.InitialSearchPhase.access$000(InitialSearchPhase.java:48) [elasticsearch-6.5.2.jar:6.5.2]
    at org.elasticsearch.action.search.InitialSearchPhase$2.innerOnResponse(InitialSearchPhase.java:217) [elasticsearch-6.5.2.jar:6.5.2]
    at org.elasticsearch.action.search.SearchActionListener.onResponse(SearchActionListener.java:45) [elasticsearch-6.5.2.jar:6.5.2]
    at org.elasticsearch.action.search.SearchActionListener.onResponse(SearchActionListener.java:29) [elasticsearch-6.5.2.jar:6.5.2]
    at org.elasticsearch.action.search.SearchExecutionStatsCollector.onResponse(SearchExecutionStatsCollector.java:68) [elasticsearch-6.5.2.jar:6.5.2]
    at org.elasticsearch.action.search.SearchExecutionStatsCollector.onResponse(SearchExecutionStatsCollector.java:36) [elasticsearch-6.5.2.jar:6.5.2]
    at org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:46) [elasticsearch-6.5.2.jar:6.5.2]
    at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleResponse(SearchTransportService.java:455) [elasticsearch-6.5.2.jar:6.5.2]
 ...
Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of org.elasticsearch.common.util.concurrent.TimedRunnable@61f80d3d on QueueResizingEsThreadPoolExecutor[name = es-data-1/search, queue capacity = 1000, min queue capacity = 1000, max queue capacity = 1000, frame size = 2000, targeted response rate = 1s, task execution EWMA = 3.4s, adjustment amount = 50, org.elasticsearch.common.util.concurrent.QueueResizingEsThreadPoolExecutor@607a1bca[Running, pool size = 4, active threads = 4, queued tasks = 1001, completed tasks = 5309]]
    at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:48) ~[elasticsearch-6.5.2.jar:6.5.2]
    at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_192]
    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) ~[?:1.8.0_192]
    at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.doExecute(EsThreadPoolExecutor.java:98) ~[elasticsearch-6.5.2.jar:6.5.2]
    ... 37 more

```

and the node gets 100% load. Do I need to bump up the thread pool queue sizes maybe?

Thanks

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 6, 2019, 7:29am UTC](https://discuss.elastic.co/t/searchphaseexecutionexception-caused-by-esrejectedexecutionexception/175775/2 "2019-05-06T07:29:48Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.