RabbitMQ river causes Java heap space error

Elastic Stack Elasticsearch
1

Hi all!
Please advice. I'm using RMQ river to post data. When a queue has near 45K messages and I create river, after maybe 20-25 seconds on 4GB of heap size I take java heap space error. I was tried to use threadpool "index" and "bulk" parameters in my cluster, but no positive results. Also tried to set up a different parameters on merge and store - the same. I reviewed a same questions at mail-lists, and src of river and seems the problem that river sends too much bulk requests. I didn't tried, but at src of a couchdb river there is a blocking queue, which seems blocks too much bulk requests.

My question is: Has anybody configure RMQ river properly with ES, or maybe I'm wrong somewhere? Thanks.

2

Hi Dennis,

I am using rabbitmq-river and have been testing it under heavy load last
week. In some situations I observed the same error. After some
investigations I discovered that source of my problems is that I have
defined very agressive analyzer. Changing it to more relaxed fixed this
problem.
I still have some issues with stability of this plugin.

On server side my configuration is:
threadpool.bulk.type: fixed
threadpool.bulk.size: 60
threadpool.bulk.queue_size: 300

Best regards,
Nickolay Kolev

14 октомври 2013, понеделник, 15:58:04 UTC+3, dennis yermakov написа:

Hi all!
Please advice. I'm using RMQ river to post data. When a queue has near 45K
messages and I create river, after maybe 20-25 seconds on 4GB of heap size
I
take java heap space error. I was tried to use threadpool "index" and
"bulk"
parameters in my cluster, but no positive results. Also tried to set up a
different parameters on merge and store - the same. I reviewed a same
questions at mail-lists, and src of river and seems the problem that river
sends too much bulk requests. I didn't tried, but at src of a couchdb
river
there is a blocking queue, which seems blocks too much bulk requests.

My question is: Has anybody configure RMQ river properly with ES, or maybe
I'm wrong somewhere? Thanks.

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/RabbitMQ-river-causes-Java-heap-space-error-tp4042511.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.