# Concurrent queries (EsRejectedExecutionException and low performance) **URL:** \ **Category:** Elasticsearch\ **Created:** [July 19, 2012, 8:56am UTC](https://discuss.elastic.co/t/concurrent-queries-esrejectedexecutionexception-and-low-performance/8462 "2012-07-19T08:56:55Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![daniel88](https://avatars.discourse-cdn.com/v4/letter/d/eb8c5e/32.png) [@daniel88](https://discuss.elastic.co/u/daniel88)\ **Post date:** [July 19, 2012, 8:56am UTC](https://discuss.elastic.co/t/concurrent-queries-esrejectedexecutionexception-and-low-performance/8462/1 "2012-07-19T08:56:55Z")
Hi, I've been testing concurrent queries, I have just one node in a server (2 \* 4 core CPU, 16G memory) and create a index (3 shards, 1 replica). I use 1000 concurrent threads to query(use TransportClient, search condition contains a termFilter and sort in a field). I've found sometimes the testing could be finished, sometimes it cound't, because there are many EsRejectedExecutionException exceptions in ES log file. Another problem is the average response time is over 2 seconds for 1000 threads (only about 80 millisencond for 10 threads), I don't know why. my thread pool setting: threadpool: search: type: blocking min: 1 size: 300 wait\_time: 30s #type: fixed #size: 80 #queue: 1000 #reject\_policy: abort index: type: blocking min: 1 size: 150 wait\_time: 30s [2012-07-20 00:00:21,753][DEBUG][action.search.type] [Cowgirl] [mail][2]: Failed to execute [org.elasticsearch.action.search.SearchRequest@5af2ee9c] while moving to second phase org.elasticsearch.common.util.concurrent.EsRejectedExecutionException at org.elasticsearch.common.util.concurrent.EsExecutors$TimedBlockingPolicy.rejectedExecution(EsExecutors.java:171) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658) at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.moveToSecondPhase(TransportSearchDfsQueryThenFetchAction.java:132) at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:228) at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$3.onResult(TransportSearchTypeAction.java:207) at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$3.onResult(TransportSearchTypeAction.java:204) at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteDfs(SearchServiceTransportAction.java:107) at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchDfsQueryThenFetchAction.java:86) at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:204) ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [July 22, 2012, 11:46pm UTC](https://discuss.elastic.co/t/concurrent-queries-esrejectedexecutionexception-and-low-performance/8462/2 "2012-07-22T23:46:37Z")
Hi, 1000 threads is quite a number of threads for just 2 4-core servers. I'm guessing the load is very high on those machines because threads are fighting for CPU cycles and there is a lot of waiting. Yo could get SPM for ES and get better insight into what's going on with performance as you increase the number of concurrent threads - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) . ## Otis Search Analytics - [Cloud Monitoring Tools & Services | Sematext](http://sematext.com/search-analytics/index.html) Scalable Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) On Thursday, July 19, 2012 4:56:55 AM UTC-4, daniel88 wrote: > Hi, > > I've been testing concurrent queries, I have just one node in a server (2 > \* > 4 core CPU, 16G memory) and create a index (3 shards, 1 replica). I use > 1000 concurrent threads to query(use TransportClient, search condition > contains a termFilter and sort in a field). I've found sometimes the > testing > could be finished, sometimes it cound't, because there are many > EsRejectedExecutionException exceptions in ES log file. > > Another problem is the average response time is over 2 seconds for 1000 > threads (only about 80 millisencond for 10 threads), I don't know why. > > my thread pool setting: > > threadpool: > search: > type: blocking > min: 1 > size: 300 > wait\_time: 30s > #type: fixed > #size: 80 > #queue: 1000 > #reject\_policy: abort > index: > type: blocking > min: 1 > size: 150 > wait\_time: 30s > > [2012-07-20 00:00:21,753][DEBUG][action.search.type] [Cowgirl] > [mail][2]: Failed to execute > [org.elasticsearch.action.search.SearchRequest@5af2ee9c] while moving to > second phase > org.elasticsearch.common.util.concurrent.EsRejectedExecutionException > at > org.elasticsearch.common.util.concurrent.EsExecutors$TimedBlockingPolicy.rejectedExecution(EsExecutors.java:171) > > ``` > at > > ``` > > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767) > > ``` > at > > ``` > > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658) > > ``` > at > > ``` > > org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.moveToSecondPhase(TransportSearchDfsQueryThenFetchAction.java:132) > > ``` > at > > ``` > > org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:228) > > ``` > at > > ``` > > org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$3.onResult(TransportSearchTypeAction.java:207) > > ``` > at > > ``` > > org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$3.onResult(TransportSearchTypeAction.java:204) > > ``` > at > > ``` > > org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteDfs(SearchServiceTransportAction.java:107) > > ``` > at > > ``` > > org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchDfsQueryThenFetchAction.java:86) > > ``` > at > > ``` > > org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:204) > > -- > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/concurrent-queries-EsRejectedExecutionException-and-low-performance-tp4020578.html](http://elasticsearch-users.115913.n3.nabble.com/concurrent-queries-EsRejectedExecutionException-and-low-performance-tp4020578.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![daniel88](https://avatars.discourse-cdn.com/v4/letter/d/eb8c5e/32.png) [@daniel88](https://discuss.elastic.co/u/daniel88)\ **Post date:** [July 24, 2012, 1:50am UTC](https://discuss.elastic.co/t/concurrent-queries-esrejectedexecutionexception-and-low-performance/8462/3 "2012-07-24T01:50:13Z")
Otis, thanks for your response. I have found the bottleneck was the IO (iowait was very high) in my previous test. I have still two probolems about ES: 1. When I do the concurrent queries ( 10/100 threads), the cpu load of ES process is very high (790%). Is there any way to limit the cpu load of ES process. 2. has\_child query is very slow. I have 2M parent docs and 2M child docs, one parent has only one child. The reponse time for only parent doc query is less than 100ms, but When I add hasChildFilter to search condition, the reponse time is greater than 20s (every query with different condition is very slow). I don't know why. curl -XPUT [http://localhost:9200/test/parent/\_mapping](http://localhost:9200/test/parent/_mapping) -d ' { "summary" : { "\_source" : {"enabled" : false}, "\_all" : {"enabled" : false}, "\_routing" : { "required" : true, "path" : "owner\_id" }, "properties" : { "message\_id" : {"type" : "string", "store" : "yes", "index":"not\_analyzed"}, "owner\_type" : {"type" : "string", "store" : "yes", "index":"not\_analyzed"}, "owner\_id" : {"type" : "string", "store" : "yes", "index":"not\_analyzed"}, "box\_id" : {"type" : "string", "store" : "yes", "index":"not\_analyzed"}, "sender" : {"type" : "string", "store" : "yes", "index":"analyzed", "index\_analyzer" : "emailaddr", "search\_analyzer" : "default"}, "receiver" : {"type" : "string", "store" : "yes", "index":"analyzed", "index\_analyzer" : "emailaddr", "search\_analyzer" : "default"}, "has\_attachement" : {"type" : "boolean", "store" : "yes"}, "size" : {"type" : "long", "store" : "yes"}, "status" : {"type" : "byte", "store" : "yes"} } } }' curl -XPUT localhost:9200/test/child/\_mapping -d ' { "activeinfo" : { "\_source" : {"enabled" : false}, "\_all" : {"enabled" : false}, "\_parent" : {"type": "summary"}, "properties" : { "content" : {"type" : "string", "store" : "yes", "index":"analyzed", "index\_analyzer" : "smartcn", "search\_analyzer" : "smartcn"}, "html\_content" : {"type" : "string", "store" : "yes", "index" : "no"}, "store\_reference" : {"type" : "string", "store" : "yes", "index":"no"} } } }' ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:19am UTC](https://discuss.elastic.co/t/concurrent-queries-esrejectedexecutionexception-and-low-performance/8462/4 "2017-07-06T03:19:22Z")