# ESRejectedExecutionException **URL:** \ **Category:** Elasticsearch\ **Created:** [April 2, 2014, 7:00am UTC](https://discuss.elastic.co/t/esrejectedexecutionexception/16759 "2014-04-02T07:00:22Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Pandiyan](https://avatars.discourse-cdn.com/v4/letter/p/3ec8ea/32.png) [@Pandiyan](https://discuss.elastic.co/u/Pandiyan)\ **Post date:** [April 2, 2014, 7:00am UTC](https://discuss.elastic.co/t/esrejectedexecutionexception/16759/1 "2014-04-02T07:00:22Z")
I've been testing concurrent queries, I have just one node in a server (1 \* 4 core CPU, 12G memory) and create a index (4 shards, 1 replica). I use 1000 concurrent threads to query(use TransportClient, search condition contains a termFilter and sort in a field). I've found sometimes the testing could be finished, sometimes it cound't, because there are many EsRejectedExecutionException exceptions in ES log file. Thread pool configuration :: # use routing concept in elasticserach-java-api. based on this entry 0 to n-1 index.number\_of\_shards: 4 index.number\_of\_replicas: 1 #Compress the data index.store.compress.stored: true # Force all memory to be locked, forcing the JVM to never swap bootstrap.mlockall: true ## Threadpool Settings # Search pool threadpool.search.type: fixed threadpool.search.size: 40 #threadpool.search.queue\_size: 100 # Bulk pool threadpool.bulk.type: fixed threadpool.bulk.size: 60 #threadpool.bulk.queue\_size: 300 # Index pool threadpool.index.type: fixed threadpool.index.size: 10 #threadpool.index.queue\_size: 100 # Indices settings indices.memory.index\_buffer\_size: 30% #indices.memory.min\_shard\_index\_buffer\_size: 12mb #indices.memory.min\_index\_buffer\_size: 96mb # Cache Sizes indices.fielddata.cache.size: 15% indices.fielddata.cache.expire: 6m indices.cache.filter.size: 15% indices.cache.filter.expire: 6m # Indexing Settings for Writes #index.refresh\_interval: 30s index.translog.flush\_threshold\_ops: 50000 ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [April 2, 2014, 4:51pm UTC](https://discuss.elastic.co/t/esrejectedexecutionexception/16759/2 "2014-04-02T16:51:54Z")
If you have 40 search threads on the node running and no queue, you should not use more than 40 search threads on the client, otherwise rejections are to be expected. Jörg On Wed, Apr 2, 2014 at 9:00 AM, Pandiyan [pandy0025@gmail.com](mailto:pandy0025@gmail.com) wrote: > I've been testing concurrent queries, I have just one node in a server (1 > > - 4 > core CPU, 12G memory) and create a index (4 shards, 1 replica). I use 1000 > concurrent threads to query(use TransportClient, search condition contains > a > termFilter and sort in a field). I've found sometimes the testing could be > finished, sometimes it cound't, because there are many > EsRejectedExecutionException exceptions in ES log file. > > Thread pool configuration :: > > # use routing concept in elasticserach-java-api. based on this entry 0 to > > n-1 > index.number\_of\_shards: 4 > index.number\_of\_replicas: 1 > > #Compress the data > index.store.compress.stored: true > > # Force all memory to be locked, forcing the JVM to never swap > > bootstrap.mlockall: true > > ## Threadpool Settings > > # Search pool > > threadpool.search.type: fixed > threadpool.search.size: 40 > #threadpool.search.queue\_size: 100 > > # Bulk pool > > threadpool.bulk.type: fixed > threadpool.bulk.size: 60 > #threadpool.bulk.queue\_size: 300 > > # Index pool > > threadpool.index.type: fixed > threadpool.index.size: 10 > #threadpool.index.queue\_size: 100 > > # Indices settings > > indices.memory.index\_buffer\_size: 30% > #indices.memory.min\_shard\_index\_buffer\_size: 12mb > #indices.memory.min\_index\_buffer\_size: 96mb > > # Cache Sizes > > indices.fielddata.cache.size: 15% > indices.fielddata.cache.expire: 6m > indices.cache.filter.size: 15% > indices.cache.filter.expire: 6m > > # Indexing Settings for Writes > > #index.refresh\_interval: 30s > index.translog.flush\_threshold\_ops: 50000 > > -- > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/ESRejectedExecutionException-tp4053295.html](http://elasticsearch-users.115913.n3.nabble.com/ESRejectedExecutionException-tp4053295.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/1396422023093-4053295.post%40n3.nabble.com](https://groups.google.com/d/msgid/elasticsearch/1396422023093-4053295.post%40n3.nabble.com) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFUSvpNE2TOWPaYavaC79BvP8JO%3DbVyyzdWUHwB34Ftng%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFUSvpNE2TOWPaYavaC79BvP8JO%3DbVyyzdWUHwB34Ftng%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![brian\_yoder](https://avatars.discourse-cdn.com/v4/letter/b/f1d935/32.png) [@brian\_yoder](https://discuss.elastic.co/u/brian_yoder)\ **Post date:** [April 3, 2014, 8:48pm UTC](https://discuss.elastic.co/t/esrejectedexecutionexception/16759/3 "2014-04-03T20:48:56Z")
This bears repeating: The LMAX Disruptor[http://lmax-exchange.github.io/disruptor/](http://lmax-exchange.github.io/disruptor/)is a an awesomely cool, small, blindingly fast, rocks-old library that is great for something like this. Create an Event class that contains your query. Create a WorkHandler class whose onEvent method is called whenever there is a query to process. Create a Disruptor object with two thread pools, a large enough ring buffer (to hold pending queries), and a MULTI\_PRODUCER strategy. Create and register a number of your WorkHandler objects, one for each thread that you want accessing Elasticsearch. This will bound the load on ES. Also set a wait strategy: For use with ES I use a BlockingWaitStrategy as a good compromise between high performance and low CPU usage (our operations folks hate to see 100% CPU even if the CPU is efficiently spinning). Then start the Disruptor and save the reference to the RingBuffer for your publisher threads. Now you can have a bazillion threads issuing queries. All they do is use the 2-phase commit logic to publish: Get a sequence number, then get an Event for that sequence, store the query into that Event, and commit the Event. Poof! The RingBuffer handles all the complexity, and is optimized for lockless performance where possible. For example, if a burst of requests come in and there are many more published events than handled events, then the RingBuffer can select a batch of events that are doled out to WorkHandler threads without any locking whatsoever. Vastly better than the constant lock/get/unlock overhead of a standard Java queue. The Disruptor adds an indistinguishably teeny tiny overhead; you won't even know it's there. When it's time to end the application, there are easy and safe shutdown methods to ensure everything is stopped without losing any queries (which could also include updates if you need them). Elasticsearch rocks. The Disruptor rocks. Put 'em together and you can blast bazillions of queries at your ES instance and never overload it. And by limiting the threads that are banging on ES, you also keep ES performing at its sweet spot and not thrashing. Regards, Brian -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/030d2ef8-f8b3-4c99-bbd4-2ad7dc606375%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/030d2ef8-f8b3-4c99-bbd4-2ad7dc606375%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:38am UTC](https://discuss.elastic.co/t/esrejectedexecutionexception/16759/4 "2017-07-06T01:38:26Z")