# Getting org.elasticsearch.common.util.concurrent.EsRejectedExecutionException **URL:** \ **Category:** Elasticsearch\ **Created:** [August 20, 2015, 7:13am UTC](https://discuss.elastic.co/t/getting-org-elasticsearch-common-util-concurrent-esrejectedexecutionexception/27731 "2015-08-20T07:13:05Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![sohilelasticsearch](https://avatars.discourse-cdn.com/v4/letter/s/ecc23a/32.png) [@sohilelasticsearch](https://discuss.elastic.co/u/sohilelasticsearch)\ **Post date:** [August 20, 2015, 7:13am UTC](https://discuss.elastic.co/t/getting-org-elasticsearch-common-util-concurrent-esrejectedexecutionexception/27731/1 "2015-08-20T07:13:05Z")
I have done setup of **elasticsearch-1.4.2** and import ~150million records from mysql to elasticsearch. Everything was working fine and suddenly after 2 days I was getting below exception, Exception is present at [http://pastebin.com/1Zmhd6UZ](http://pastebin.com/1Zmhd6UZ) location. I have only one data node and everything is running with default Elastic Search configuration. I have written one scheduler which runs every 6 hours and insert 0.5 million records to elasticsearch. Please suggest me how to solve this issue. ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [August 21, 2015, 3:31pm UTC](https://discuss.elastic.co/t/getting-org-elasticsearch-common-util-concurrent-esrejectedexecutionexception/27731/2 "2015-08-21T15:31:07Z")
This is the way that elasticsearch tells you that you are sending data too quickly compared to what it can ingest. When you get such an exception, you should slow down the ingestion rate a bit. ---
**Author:** ![sohilelasticsearch](https://avatars.discourse-cdn.com/v4/letter/s/ecc23a/32.png) [@sohilelasticsearch](https://discuss.elastic.co/u/sohilelasticsearch)\ **Post date:** [August 22, 2015, 6:21am UTC](https://discuss.elastic.co/t/getting-org-elasticsearch-common-util-concurrent-esrejectedexecutionexception/27731/3 "2015-08-22T06:21:54Z")
Changing ingestion rates means amount of data to insert into elasticsearch? ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [August 24, 2015, 7:51am UTC](https://discuss.elastic.co/t/getting-org-elasticsearch-common-util-concurrent-esrejectedexecutionexception/27731/4 "2015-08-24T07:51:09Z")
Yes. This message means that you are sending too much data in parallel compared to what elasticsearch can handle, so you need to reduce the number of threads/workers that send data to elasticsearch, at least temporarily. ---
**Author:** ![sohilelasticsearch](https://avatars.discourse-cdn.com/v4/letter/s/ecc23a/32.png) [@sohilelasticsearch](https://discuss.elastic.co/u/sohilelasticsearch)\ **Post date:** [August 24, 2015, 3:02pm UTC](https://discuss.elastic.co/t/getting-org-elasticsearch-common-util-concurrent-esrejectedexecutionexception/27731/5 "2015-08-24T15:02:27Z")
Can you please help me in below understanding, while importing **~150** **million** data from mysql to elasticsearch I didn't get this error whereas during scheduler run I am getting this error. Any specific reason for this? ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [August 24, 2015, 3:15pm UTC](https://discuss.elastic.co/t/getting-org-elasticsearch-common-util-concurrent-esrejectedexecutionexception/27731/6 "2015-08-24T15:15:02Z")
If you did the indexing in similar conditions then this is a bit surprising. ---
**Author:** ![sohilelasticsearch](https://avatars.discourse-cdn.com/v4/letter/s/ecc23a/32.png) [@sohilelasticsearch](https://discuss.elastic.co/u/sohilelasticsearch)\ **Post date:** [August 24, 2015, 3:45pm UTC](https://discuss.elastic.co/t/getting-org-elasticsearch-common-util-concurrent-esrejectedexecutionexception/27731/7 "2015-08-24T15:45:06Z")
Here is the way I am doing the indexing, A. To Fetch ~150 million records curl -XPUT '[http://XXX.X.X.XXX:9200/\_river/blk\_jdbc\_river/\_meta](http://XXX.X.X.XXX:9200/_river/blk_jdbc_river/_meta)' -d '{ "type" : "jdbc", "jdbc" : { "strategy": "simple", "autocommit": true, "driver" : "com.mysql.jdbc.Driver", "url" : "jdbc:mysql://localhost:3306/KA", "user" : "**_", "password" : "_**\*\*", "sql" : [ { "statement" : "select id as "\_id", p\_id as "pid" ,s\_id as "sid" ,d\_name as "dname" from sohil" } ], "maxbulkactions":5000, "maxconcurrrentbulkactions":1, "index" : "blks", "type" : "blk", "type\_mapping": {"blks" : {"properties" : {"\_id":{"type":"long","store":"yes"},"dname":{"type":"string","store":"yes","index":"not\_analyzed"},"pid":{"type":"long","store":"yes"},"sid":{"type":"long","store":"yes"}}}} } }' B. To fetch delta at every run (Frequency :- every 6 hours) curl -XPUT '[http://XXX.X.X.XXX:9200/\_river/update\_blk\_jdbc\_river/\_meta](http://XXX.X.X.XXX:9200/_river/update_blk_jdbc_river/_meta)' -d '{ "type" : "jdbc", "jdbc" : { "strategy": "simple", "autocommit": true, "schedule" : "0 _/6 \* \* \* ?", "driver" : "com.mysql.jdbc.Driver", "url" : "jdbc:mysql://localhost:3306/KA", "user" : "_**", "password" : "**\*\*\*", "maxbulkactions":5000, "maxconcurrrentbulkactions":1, "sql" : [ { "statement" : "select id as "\_id", p\_id as "pid" ,s\_id as "sid" ,d\_name as "dname" from sohil where id \> XXX" } ] , "index" : "blks", "type" : "blk", "type\_mapping": {"blks" : {"properties" : {"\_id":{"type":"long","store":"yes"},"dname":{"type":"string","store":"yes","index":"not\_analyzed"},"pid":{"type":"long","store":"yes"},"sid":{"type":"long","store":"yes"}}}} } }' Please let me know in case I am doing anything wrong. ---
**Author:** ![Srinath\_C](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/srinath_c/32/48806_2.png) [@Srinath\_C](https://discuss.elastic.co/u/Srinath_C)\ **Post date:** [August 26, 2015, 2:03am UTC](https://discuss.elastic.co/t/getting-org-elasticsearch-common-util-concurrent-esrejectedexecutionexception/27731/8 "2015-08-26T02:03:57Z")
I see that the rejected execution exception is due to search threadpool queue exceeding 1000. What kind of queries are running through the system? ---
**Author:** ![sohilelasticsearch](https://avatars.discourse-cdn.com/v4/letter/s/ecc23a/32.png) [@sohilelasticsearch](https://discuss.elastic.co/u/sohilelasticsearch)\ **Post date:** [August 27, 2015, 4:49pm UTC](https://discuss.elastic.co/t/getting-org-elasticsearch-common-util-concurrent-esrejectedexecutionexception/27731/9 "2015-08-27T16:49:49Z")
Hi @Srinath_C, There are three types of queries. 1. Search query (through java client) 2. Delete query(through elasticsearch, mark records of elasticsearch to delete) 3. Insert query(Through scheduler) I forgot to mention that there is one more scheduler, which runs every 5 hour and delete records from elasticsearch. Below is the river configuration for deleting records from elasticsearch, curl -XPUT '[http://XX.XXX.XXX.XX:9200/\_river/delete\_blklist\_jdbc\_river/\_meta](http://XX.XXX.XXX.XX:9200/_river/delete_blklist_jdbc_river/_meta)' -d '{ "type":"jdbc", "jdbc":{ "strategy":"simple", "schedule":"0 0 0/5 \* \* ?", "autocommit":true, "driver":"com.mysql.jdbc.Driver", "url":"jdbc:mysql://XX.XXX.XXX.XX:3306/KA", "user":"XXXX", "password":"XXXXXX", "maxbulkactions":5000, "maxconcurrrentbulkactions":1, "sql":"select id as "\_id", "delete" as "\_optype", "blks" as "\_index", "blk" as "\_type" from deleted\_id\_table" } }' ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [August 28, 2015, 4:31pm UTC](https://discuss.elastic.co/t/getting-org-elasticsearch-common-util-concurrent-esrejectedexecutionexception/27731/10 "2015-08-28T16:31:57Z")
The message ```auto org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution (queue capacity 1000) on org.elasticsearch.search.action.SearchServiceTransportAction$23@220dbcbd ``` is not directly related to JDBC river, because JDBC river does not execute search requests at all. You should find out who is continuing to send search requests to your cluster. Also, you have to find if the search requests responded with useful results in reasonable time, or if they time out. If they time out, search requests can fill up the search queue quite soon, within seconds, even when the size is 1000. Most probably, cluster state has switched, or search can not continue due to cluster or index blocks, or other incidents. ---
**Author:** ![sohilelasticsearch](https://avatars.discourse-cdn.com/v4/letter/s/ecc23a/32.png) [@sohilelasticsearch](https://discuss.elastic.co/u/sohilelasticsearch)\ **Post date:** [August 31, 2015, 5:24am UTC](https://discuss.elastic.co/t/getting-org-elasticsearch-common-util-concurrent-esrejectedexecutionexception/27731/11 "2015-08-31T05:24:22Z")
Hi @jprante, Thanks for your answer. I will try to find out the root cause. Is there any way to recover from such type of exception? Thanks, Sohil ---
**Author:** ![sohilelasticsearch](https://avatars.discourse-cdn.com/v4/letter/s/ecc23a/32.png) [@sohilelasticsearch](https://discuss.elastic.co/u/sohilelasticsearch)\ **Post date:** [September 5, 2015, 5:34am UTC](https://discuss.elastic.co/t/getting-org-elasticsearch-common-util-concurrent-esrejectedexecutionexception/27731/12 "2015-09-05T05:34:00Z")
Hi, I have identified and stopped the one who was continuously sending the search request. After that I have rebuild the elasticsearch index, but after two days I started getting the same exception. Here is the output of cluster and index status [http://pastebin.com/c56wZmEf](http://pastebin.com/c56wZmEf) It will be great if someone can help me to resolve this. ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [September 5, 2015, 8:28am UTC](https://discuss.elastic.co/t/getting-org-elasticsearch-common-util-concurrent-esrejectedexecutionexception/27731/13 "2015-09-05T08:28:15Z")
You do not need to rebuild the index only because of a search exception. The information you give is not sufficient. You should post a copy of the exception you get now, the configuration of your cluster, of your index, and what the identified search request looks like. ---
**Author:** ![sohilelasticsearch](https://avatars.discourse-cdn.com/v4/letter/s/ecc23a/32.png) [@sohilelasticsearch](https://discuss.elastic.co/u/sohilelasticsearch)\ **Post date:** [September 7, 2015, 6:30am UTC](https://discuss.elastic.co/t/getting-org-elasticsearch-common-util-concurrent-esrejectedexecutionexception/27731/14 "2015-09-07T06:30:53Z")
Hi @jprante, Here is the exception which I got now [http://pastebin.com/xwAEZDqn](http://pastebin.com/xwAEZDqn) When I hit normal search call, I got curl -XGET 'XX.XXX.XXX.XX:9200/blks/\_search?pretty&q=dname:2010.census.gov' { "error" : "SearchPhaseExecutionException[Failed to execute phase [query], all shards failed; shardFailures {[-y2z8tICS\_6mywTPw77x2A][blks][0]: EsRejectedExecutionException[rejected execution (queue capacity 1000) on org.elasticsearch.search.action.SearchServiceTransportAction$23@49cb7fbc]}{[-y2z8tICS\_6mywTPw77x2A][blks][1]: EsRejectedExecutionException[rejected execution (queue capacity 1000) on org.elasticsearch.search.action.SearchServiceTransportAction$23@4e781ac0]}{[-y2z8tICS\_6mywTPw77x2A][blks][2]: EsRejectedExecutionException[rejected execution (queue capacity 1000) on org.elasticsearch.search.action.SearchServiceTransportAction$23@7bf30706]}{[-y2z8tICS\_6mywTPw77x2A][blks][3]: EsRejectedExecutionException[rejected execution (queue capacity 1000) on org.elasticsearch.search.action.SearchServiceTransportAction$23@eb2ccea]}{[-y2z8tICS\_6mywTPw77x2A][blks][4]: EsRejectedExecutionException[rejected execution (queue capacity 1000) on org.elasticsearch.search.action.SearchServiceTransportAction$23@464adf9a]}]", "status" : 429 } Can you please point me to the API through which I can get information about the configuration of cluster, of index ? Regarding search request, do you want the actual search query which I am hitting to fetch data from ES? Thanks, Sohil ---
**Author:** ![sohilelasticsearch](https://avatars.discourse-cdn.com/v4/letter/s/ecc23a/32.png) [@sohilelasticsearch](https://discuss.elastic.co/u/sohilelasticsearch)\ **Post date:** [September 9, 2015, 4:40am UTC](https://discuss.elastic.co/t/getting-org-elasticsearch-common-util-concurrent-esrejectedexecutionexception/27731/15 "2015-09-09T04:40:36Z")
Hi @jprante, I am getting the same exception after every two days. Is it because of only single elastic search node? ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [September 12, 2015, 11:36am UTC](https://discuss.elastic.co/t/getting-org-elasticsearch-common-util-concurrent-esrejectedexecutionexception/27731/16 "2015-09-12T11:36:51Z")
> [@sohilelasticsearch](#): > > I am getting the same exception after every two days. Is it because of only single Elasticsearch node? You mean you only use a single node??? From the logs you published at [ElasticSearch Exception log file - Pastebin.com](http://pastebin.com/xwAEZDqn) it is quite clear your cluster runs out of resources. You need to ramp up more resources or you should add more nodes to your cluster to get more heap and CPU power. ---
**Author:** ![sohilelasticsearch](https://avatars.discourse-cdn.com/v4/letter/s/ecc23a/32.png) [@sohilelasticsearch](https://discuss.elastic.co/u/sohilelasticsearch)\ **Post date:** [September 12, 2015, 12:35pm UTC](https://discuss.elastic.co/t/getting-org-elasticsearch-common-util-concurrent-esrejectedexecutionexception/27731/17 "2015-09-12T12:35:32Z")
Hi @jprante, I am using only one elasticsearch node. My mapping is { "blks": { "mappings": { "blk": { "properties": { "dname": { "type": "string" }, "pid": { "type": "long" }, "sid": { "type": "long" } } } } } } and my **dname** field is **notanalyzed**. Below are only two types of query which I am executing on my elasticsearch node. 1. Search on **dname** field :- (exact match) OR (ends with) [2015-09-07 22:56:19,554][TRACE][index.search.slowlog.query] [Plug] [blks][0] took[39.6ms], took\_millis[39], types[blk], stats[], search\_type[DFS\_QUERY\_THEN\_FETCH], total\_shards[5], source[{"from":0,"size":750000,"query":{"bool":{"must":{"query\_string":{"query":"[abc.com](http://abc.com)","fields":["dname"]}},"should":{"query\_string":{"query":"\*.abc.com","fields":["dname"]}}}},"post\_filter":{"or":{"filters":[]}}}], extra\_source[], 1. Search on **pid** and **sid** field :- Query :- pid=X AND sid=Y [2015-09-07 22:56:19,554][TRACE][index.search.slowlog.query] [Plug] [blks][1] took[7.1ms], took\_millis[7], types[blk], stats[], search\_type[DFS\_QUERY\_THEN\_FETCH], total\_shards[5], source[{"from":0,"size":750000,"query":{"bool":{"must":[{"query\_string":{"query":"504","fields":["pid"]}},{"query\_string":{"query":"0","fields":["sid"]}}]}}}], extra\_source[], Please let me know in case you need more details. Currently I am using **elasticsearch-1.7.1** on production. Your suggestion are always welcome. ---
**Author:** ![Srinath\_C](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/srinath_c/32/48806_2.png) [@Srinath\_C](https://discuss.elastic.co/u/Srinath_C)\ **Post date:** [September 14, 2015, 1:52am UTC](https://discuss.elastic.co/t/getting-org-elasticsearch-common-util-concurrent-esrejectedexecutionexception/27731/18 "2015-09-14T01:52:14Z")
@sohilelasticsearch did you try installing marvel and looking into the queue size and other parameters? Definitely worth the try. You could also try adjusting "maxbulkactions":5000 and maxconcurrrentbulkactions":1 parameters. ---
**Author:** ![sohilelasticsearch](https://avatars.discourse-cdn.com/v4/letter/s/ecc23a/32.png) [@sohilelasticsearch](https://discuss.elastic.co/u/sohilelasticsearch)\ **Post date:** [September 14, 2015, 12:17pm UTC](https://discuss.elastic.co/t/getting-org-elasticsearch-common-util-concurrent-esrejectedexecutionexception/27731/19 "2015-09-14T12:17:55Z")
@Srinath_C I have already set "maxbulkactions":5000 and maxconcurrrentbulkactions":1. ---
**Author:** ![Maxim\_Valyanskiy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/maxim_valyanskiy/32/787_2.png) [@Maxim\_Valyanskiy](https://discuss.elastic.co/u/Maxim_Valyanskiy)\ **Post date:** [September 14, 2015, 8:52pm UTC](https://discuss.elastic.co/t/getting-org-elasticsearch-common-util-concurrent-esrejectedexecutionexception/27731/20 "2015-09-14T20:52:29Z")
Elasticsearch is not good at executing queries that you make. I have few ideas that can improve performance: 1. Do not query for 750000 rows. Use search/scroll request. If order (score sorting) is not required, use scan/scroll request. 2. Again, if score sorting is not required, use "term filter" to run exact match. Wrap it in filtered query to put in search request. 3. Wildcard suffix search is expensive operation. I suggest you to use path hierarchy tokenizer with reverse order and '.' as a separator. With this settings you could run term filter/query to match your domains. [Next page](https://discuss.elastic.co/t/getting-org-elasticsearch-common-util-concurrent-esrejectedexecutionexception/27731.md?page=2)