ESRejectedExecutionException

Pandiyan · April 2, 2014, 7:00am

I've been testing concurrent queries, I have just one node in a server (1 * 4 core CPU, 12G memory) and create a index (4 shards, 1 replica). I use 1000 concurrent threads to query(use TransportClient, search condition contains a termFilter and sort in a field). I've found sometimes the testing could be finished, sometimes it cound't, because there are many EsRejectedExecutionException exceptions in ES log file.

Thread pool configuration ::

use routing concept in elasticserach-java-api. based on this entry 0 to n-1

index.number_of_shards: 4
index.number_of_replicas: 1

#Compress the data
index.store.compress.stored: true

Force all memory to be locked, forcing the JVM to never swap

bootstrap.mlockall: true

Threadpool Settings

Search pool

threadpool.search.type: fixed
threadpool.search.size: 40
#threadpool.search.queue_size: 100

Bulk pool

threadpool.bulk.type: fixed
threadpool.bulk.size: 60
#threadpool.bulk.queue_size: 300

Index pool

threadpool.index.type: fixed
threadpool.index.size: 10
#threadpool.index.queue_size: 100

Indices settings

indices.memory.index_buffer_size: 30%
#indices.memory.min_shard_index_buffer_size: 12mb
#indices.memory.min_index_buffer_size: 96mb

Cache Sizes

indices.fielddata.cache.size: 15%
indices.fielddata.cache.expire: 6m
indices.cache.filter.size: 15%
indices.cache.filter.expire: 6m

Indexing Settings for Writes

#index.refresh_interval: 30s
index.translog.flush_threshold_ops: 50000

jprante · April 2, 2014, 4:51pm

If you have 40 search threads on the node running and no queue, you should
not use more than 40 search threads on the client, otherwise rejections are
to be expected.

Jörg

On Wed, Apr 2, 2014 at 9:00 AM, Pandiyan pandy0025@gmail.com wrote:

I've been testing concurrent queries, I have just one node in a server (1

4
core CPU, 12G memory) and create a index (4 shards, 1 replica). I use 1000
concurrent threads to query(use TransportClient, search condition contains
a
termFilter and sort in a field). I've found sometimes the testing could be
finished, sometimes it cound't, because there are many
EsRejectedExecutionException exceptions in ES log file.

Thread pool configuration ::

use routing concept in elasticserach-java-api. based on this entry 0 to

n-1
index.number_of_shards: 4
index.number_of_replicas: 1

#Compress the data
index.store.compress.stored: true

Force all memory to be locked, forcing the JVM to never swap

bootstrap.mlockall: true

Threadpool Settings

Search pool

threadpool.search.type: fixed
threadpool.search.size: 40
#threadpool.search.queue_size: 100

Bulk pool

threadpool.bulk.type: fixed
threadpool.bulk.size: 60
#threadpool.bulk.queue_size: 300

Index pool

threadpool.index.type: fixed
threadpool.index.size: 10
#threadpool.index.queue_size: 100

Indices settings

indices.memory.index_buffer_size: 30%
#indices.memory.min_shard_index_buffer_size: 12mb
#indices.memory.min_index_buffer_size: 96mb

Cache Sizes

indices.fielddata.cache.size: 15%
indices.fielddata.cache.expire: 6m
indices.cache.filter.size: 15%
indices.cache.filter.expire: 6m

Indexing Settings for Writes

#index.refresh_interval: 30s
index.translog.flush_threshold_ops: 50000

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/ESRejectedExecutionException-tp4053295.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1396422023093-4053295.post%40n3.nabble.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFUSvpNE2TOWPaYavaC79BvP8JO%3DbVyyzdWUHwB34Ftng%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

brian_yoder · April 3, 2014, 8:48pm

This bears repeating: The LMAX Disruptorhttp://lmax-exchange.github.io/disruptor/is a an awesomely cool, small, blindingly fast, rocks-old library that is
great for something like this.

Create an Event class that contains your query.

Create a WorkHandler class whose onEvent method is called whenever
there is a query to process.

Create a Disruptor object with two thread pools, a large enough ring
buffer (to hold pending queries), and a MULTI_PRODUCER strategy. Create and
register a number of your WorkHandler objects, one for each thread that you
want accessing Elasticsearch. This will bound the load on ES. Also set a
wait strategy: For use with ES I use a BlockingWaitStrategy as a good
compromise between high performance and low CPU usage (our operations folks
hate to see 100% CPU even if the CPU is efficiently spinning).

Then start the Disruptor and save the reference to the RingBuffer for your
publisher threads.

Now you can have a bazillion threads issuing queries. All they do is use
the 2-phase commit logic to publish: Get a sequence number, then get an
Event for that sequence, store the query into that Event, and commit the
Event. Poof!

The RingBuffer handles all the complexity, and is optimized for lockless
performance where possible. For example, if a burst of requests come in and
there are many more published events than handled events, then the
RingBuffer can select a batch of events that are doled out to WorkHandler
threads without any locking whatsoever. Vastly better than the constant
lock/get/unlock overhead of a standard Java queue. The Disruptor adds an
indistinguishably teeny tiny overhead; you won't even know it's there.

When it's time to end the application, there are easy and safe shutdown
methods to ensure everything is stopped without losing any queries (which
could also include updates if you need them).

Elasticsearch rocks. The Disruptor rocks. Put 'em together and you can
blast bazillions of queries at your ES instance and never overload it. And
by limiting the threads that are banging on ES, you also keep ES performing
at its sweet spot and not thrashing.

Regards,
Brian

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/030d2ef8-f8b3-4c99-bbd4-2ad7dc606375%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.