EsRejectedExecutionException[rejected execution (queue capacity 50)


(Ramchandra Phadake) #1

Hi,

Recently we migrated from 0.90.1 to 0.90.9. We did change
index.codec.bloom.load=false initially but even with true we obsetve same
behaviour.
Nothing else has changed from indexing code. We are using bulk API. Any
pointers why we can get EsRejectedExecutionException?
This doesn't frequently in indexing process.

We did try by setting with few thousand size but this exception comes but
rarely.
After setting bulk queue to -1 it appears to be working. -1 means unbounded
size so is it safe to set to -1.

Thanks,
Ram

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/89c6e465-c23f-4234-9667-cd09be4a7178%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Alexander Reelsen) #2

Hey,

can you explain what kind of problems you have? And why you think that not
loading bloom filters will solve it?

Setting a queue to unbounded might mean, that you possibly can totally
overload your system.
First, you should use the nodes stats APIs to monitor your thread bools,
specifically the bulk one. Maybe changing its limits might make more sense
instead of setting it to unbounded. Just changing some stuff without
monitoring first, might lead to setting the wrong configuration options and
maybe improve things, or making them worse.

--Alex

On Fri, Jan 31, 2014 at 6:48 AM, Ramchandra Phadake <
ramchandra.phadake@gmail.com> wrote:

Hi,

Recently we migrated from 0.90.1 to 0.90.9. We did change
index.codec.bloom.load=false initially but even with true we obsetve same
behaviour.
Nothing else has changed from indexing code. We are using bulk API. Any
pointers why we can get EsRejectedExecutionException?
This doesn't frequently in indexing process.

We did try by setting with few thousand size but this exception comes but
rarely.
After setting bulk queue to -1 it appears to be working. -1 means
unbounded size so is it safe to set to -1.

Thanks,
Ram

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/89c6e465-c23f-4234-9667-cd09be4a7178%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGCwEM9jkioPjuHpEfu-GWn_rkkCrniDWVyHwXCbaG3HKDxejw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Jörg Prante) #3

The new setting of bounded bulk queue is a precaution not to overwhelm the
internal ES buffers.

Unbounded queue size is potentially dangerous since your node operations
may get stalled by too many bulk requests.

You could increase bulk queue_size from 50 to 100 if that works better.
But, the soundest method is to check your bulk indexing code for capping
the number of concurrent bulk requests. Or you may increase the number of
nodes/shards. Right now it seems your concurrency setting is too high for
the capacity of your cluster (just a wild guess, without knowing your
numbers).

Jörg

On Fri, Jan 31, 2014 at 6:48 AM, Ramchandra Phadake <
ramchandra.phadake@gmail.com> wrote:

Hi,

Recently we migrated from 0.90.1 to 0.90.9. We did change
index.codec.bloom.load=false initially but even with true we obsetve same
behaviour.
Nothing else has changed from indexing code. We are using bulk API. Any
pointers why we can get EsRejectedExecutionException?
This doesn't frequently in indexing process.

We did try by setting with few thousand size but this exception comes but
rarely.
After setting bulk queue to -1 it appears to be working. -1 means
unbounded size so is it safe to set to -1.

Thanks,
Ram

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/89c6e465-c23f-4234-9667-cd09be4a7178%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoExvP3W-6MiR%3DSG_c%2BR9kJz_koLLq8zOF_JML_axhrQbw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Ramchandra Phadake) #4

Thanks Alex & Jorg.

can you explain what kind of problems you have?
The problem is getting EsRejectedExecutionException during bulk indexing.
This error started appearing after our move to 0.90.9.
We wanted to save on memory so have set bloom filter settings as false
initially. In fact we migrated to 0.90.9 for this change only.

And why you think that not loading bloom filters will solve it?
Thinking was with bloom filter false the indexing time will increase and
that may fill up the queue fast so tried with true as well.
But the error is coming independent of this setting.

yes checking with
http://localhost:9200/_nodes/thread_pool/stats?pretty=true for knowing the
bulk rejections.

the soundest method is to check your bulk indexing code for capping the
number of concurrent bulk requests. Or you may increase the number of
nodes/shards. > Right now it seems your concurrency setting is too high for
the capacity of your cluster (just a wild guess, without knowing your
numbers)

out setup:
4 shards per index. max 150 indices or so & have single node setup.
In our case 1 bulk call has 2000 entries to be indexed & 4 concurrent bulk
calls.

Will also reduce bulk entries size and max 3 concurrent bulk calls and
check on this again to see if that helps. Thanks.
Btw since current indexing rate was working fine with 0.90.1 this got
introduced with 0.90.9 for sure against our bulk indexing rate.

Thanks,
Ram

On Monday, February 3, 2014 4:35:49 PM UTC+5:30, Jörg Prante wrote:

The new setting of bounded bulk queue is a precaution not to overwhelm the
internal ES buffers.

Unbounded queue size is potentially dangerous since your node operations
may get stalled by too many bulk requests.

You could increase bulk queue_size from 50 to 100 if that works better.
But, the soundest method is to check your bulk indexing code for capping
the number of concurrent bulk requests. Or you may increase the number of
nodes/shards. Right now it seems your concurrency setting is too high for
the capacity of your cluster (just a wild guess, without knowing your
numbers).

Jörg

On Fri, Jan 31, 2014 at 6:48 AM, Ramchandra Phadake <
ramchandr...@gmail.com <javascript:>> wrote:

Hi,

Recently we migrated from 0.90.1 to 0.90.9. We did change
index.codec.bloom.load=false initially but even with true we obsetve same
behaviour.
Nothing else has changed from indexing code. We are using bulk API. Any
pointers why we can get EsRejectedExecutionException?
This doesn't frequently in indexing process.

We did try by setting with few thousand size but this exception comes but
rarely.
After setting bulk queue to -1 it appears to be working. -1 means
unbounded size so is it safe to set to -1.

Thanks,
Ram

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/89c6e465-c23f-4234-9667-cd09be4a7178%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/17fabc2a-7cda-4e15-8ec3-eb6662ca5ca8%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Klavs Klavsen) #5

did you resolve this issue?.. we seem top have the same issue - also with
0.90.9

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/06c6e0f2-e20c-462a-9759-d3d71fcf171c%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Le Thanh Nguyen) #6

Hi,

we have the same problem with ES 1.1.0

Increase only queue_size could resolve the problem? Anything else?

We have 30 concurrent bulk but not on the same document, each bulk contains
about 30 documents.

See also
: https://github.com/richardwilly98/elasticsearch-river-mongodb/issues/223

On Friday, January 31, 2014 6:48:31 AM UTC+1, Ramchandra Phadake wrote:

Hi,

Recently we migrated from 0.90.1 to 0.90.9. We did change
index.codec.bloom.load=false initially but even with true we obsetve same
behaviour.
Nothing else has changed from indexing code. We are using bulk API. Any
pointers why we can get EsRejectedExecutionException?
This doesn't frequently in indexing process.

We did try by setting with few thousand size but this exception comes but
rarely.
After setting bulk queue to -1 it appears to be working. -1 means
unbounded size so is it safe to set to -1.

Thanks,
Ram

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/bb90c1f7-7086-4f99-a892-98dd0b3ce957%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(deer) #7

using cached for both index and search thread pool?

curl -XPUT localhost:9200/_cluster/settings -d '{
"persistent" : {
"threadpool.index.type" : "cached",
"threadpool.search.type" : "cached"
}
}'

On Monday, April 28, 2014 5:40:14 PM UTC+8, Le Thanh Nguyen wrote:

Hi,

we have the same problem with ES 1.1.0

Increase only queue_size could resolve the problem? Anything else?

We have 30 concurrent bulk but not on the same document, each bulk
contains about 30 documents.

See also :
https://github.com/richardwilly98/elasticsearch-river-mongodb/issues/223

On Friday, January 31, 2014 6:48:31 AM UTC+1, Ramchandra Phadake wrote:

Hi,

Recently we migrated from 0.90.1 to 0.90.9. We did change
index.codec.bloom.load=false initially but even with true we obsetve same
behaviour.
Nothing else has changed from indexing code. We are using bulk API. Any
pointers why we can get EsRejectedExecutionException?
This doesn't frequently in indexing process.

We did try by setting with few thousand size but this exception comes but
rarely.
After setting bulk queue to -1 it appears to be working. -1 means
unbounded size so is it safe to set to -1.

Thanks,
Ram

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2233852a-2afe-4eb0-b0cf-6816c6e84602%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(srikanth ramineni) #8

Hi ,

I am getting below es rejection exception.
problem updating content indexing for entity:
85539735340578996965234585294218135410438591031260132420 error:
EsRejectedExecutionException[rejected execution (queue capacity 50)
on
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@4a444d2e]

Can you please provide me code how to set bulk queue to -1 .
Help would be appreciable.

Thanks,
Srikanth.

On Thursday, 30 January 2014 21:48:31 UTC-8, Ramchandra Phadake wrote:

Hi,

Recently we migrated from 0.90.1 to 0.90.9. We did change
index.codec.bloom.load=false initially but even with true we obsetve same
behaviour.
Nothing else has changed from indexing code. We are using bulk API. Any
pointers why we can get EsRejectedExecutionException?
This doesn't frequently in indexing process.

We did try by setting with few thousand size but this exception comes but
rarely.
After setting bulk queue to -1 it appears to be working. -1 means
unbounded size so is it safe to set to -1.

Thanks,
Ram

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/da25b26f-d168-4733-8b6e-f1f42c1fac7c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Ivan Brusic) #9

The queue sizes are defined in the threadpool section of the configuration.
IIRC, they cannot be updated dynamically:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-threadpool.html

--
Ivan

On Fri, May 30, 2014 at 2:56 PM, srikanth ramineni ri.srikanth@gmail.com
wrote:

Hi ,

I am getting below es rejection exception.
problem updating content indexing for entity:
85539735340578996965234585294218135410438591031260132420 error:
EsRejectedExecutionException[rejected execution (queue capacity 50)
on
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@4a444d2e
]

Can you please provide me code how to set bulk queue to -1 .
Help would be appreciable.

Thanks,
Srikanth.

On Thursday, 30 January 2014 21:48:31 UTC-8, Ramchandra Phadake wrote:

Hi,

Recently we migrated from 0.90.1 to 0.90.9. We did change
index.codec.bloom.load=false initially but even with true we obsetve same
behaviour.
Nothing else has changed from indexing code. We are using bulk API. Any
pointers why we can get EsRejectedExecutionException?
This doesn't frequently in indexing process.

We did try by setting with few thousand size but this exception comes but
rarely.
After setting bulk queue to -1 it appears to be working. -1 means
unbounded size so is it safe to set to -1.

Thanks,
Ram

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/da25b26f-d168-4733-8b6e-f1f42c1fac7c%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/da25b26f-d168-4733-8b6e-f1f42c1fac7c%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQCNNNA55zkWx9p-SS3ctZjyX4q%2BgqYgLfHfyF4OTmaN2Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(system) #10