Queue capacity and EsRejectedExecutionException leads to loss of data

Elastic Stack Elasticsearch
1

Hi,

We have installed the RabbitMQ river plugin to pull data from our Queue and
adding them to ES. The thing is that at some point we are receiving the
following exception and we have as a result to lose data.

[1775]: index [events-idx], type [click], id

[3f6e4604146b435aabcf4ea5a493fd32], message
[EsRejectedExecutionException[rejected execution (queue capacity 50) on
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@12843ca2]]

We have changed the configuration of queue size to 1000 and the problem
disappeared.

My question is that is there any configuration/way to tell ES to instead of
throwing this exception and discarding the document to wait for available
resources (with the corresponding performance impact)?

Thanks

Thomas

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4e11ce4e-26d4-4a53-a53e-0ba89bde4605%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

I think you should open an issue.

It's to me somehow related to this issue: https://github.com/elasticsearch/elasticsearch-river-rabbitmq/issues/47
and its pending PR: https://github.com/elasticsearch/elasticsearch-river-rabbitmq/pull/48

I think we should have a new option to let the user decides if if want to ack messages in case of failure or leave them in queue.

My 2 cents

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 26 février 2014 à 14:49:00, Thomas (thomas.bolis@gmail.com) a écrit:

Hi,

We have installed the RabbitMQ river plugin to pull data from our Queue and adding them to ES. The thing is that at some point we are receiving the following exception and we have as a result to lose data.

[1775]: index [events-idx], type [click], id [3f6e4604146b435aabcf4ea5a493fd32], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@12843ca2]]

We have changed the configuration of queue size to 1000 and the problem disappeared.

My question is that is there any configuration/way to tell ES to instead of throwing this exception and discarding the document to wait for available resources (with the corresponding performance impact)?

Thanks

Thomas

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4e11ce4e-26d4-4a53-a53e-0ba89bde4605%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.530dfa41.22221a70.5e46%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/groups/opt_out.

3

Thanks David,

So this is a rabbitMQRiver issue, is there a need to open a separate issue?
(Never done the procedure, will look this one)

Thomas

On Wednesday, 26 February 2014 15:48:55 UTC+2, Thomas wrote:

Hi,

We have installed the RabbitMQ river plugin to pull data from our Queue
and adding them to ES. The thing is that at some point we are receiving the
following exception and we have as a result to lose data.

[1775]: index [events-idx], type [click], id

[3f6e4604146b435aabcf4ea5a493fd32], message
[EsRejectedExecutionException[rejected execution (queue capacity 50) on
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@12843ca2]]

We have changed the configuration of queue size to 1000 and the problem
disappeared.

My question is that is there any configuration/way to tell ES to instead
of throwing this exception and discarding the document to wait for
available resources (with the corresponding performance impact)?

Thanks

Thomas

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/03dcb0ea-2b6a-478b-b678-f52ecbc09298%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

I think that adding a comment into the existing issue would be fine.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 26 février 2014 à 17:00:12, Thomas (thomas.bolis@gmail.com) a écrit:

Thanks David,

So this is a rabbitMQRiver issue, is there a need to open a separate issue? (Never done the procedure, will look this one)

Thomas

On Wednesday, 26 February 2014 15:48:55 UTC+2, Thomas wrote:
Hi,

We have installed the RabbitMQ river plugin to pull data from our Queue and adding them to ES. The thing is that at some point we are receiving the following exception and we have as a result to lose data.

[1775]: index [events-idx], type [click], id [3f6e4604146b435aabcf4ea5a493fd32], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@12843ca2]]

We have changed the configuration of queue size to 1000 and the problem disappeared.

My question is that is there any configuration/way to tell ES to instead of throwing this exception and discarding the document to wait for available resources (with the corresponding performance impact)?

Thanks

Thomas

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/03dcb0ea-2b6a-478b-b678-f52ecbc09298%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.530e1318.440badfc.5e46%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/groups/opt_out.