# RabbitMQ river & bulksize **URL:** \ **Category:** Elasticsearch\ **Created:** [April 25, 2012, 8:54am UTC](https://discuss.elastic.co/t/rabbitmq-river-bulksize/7463 "2012-04-25T08:54:43Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Max\_Kossatz](https://avatars.discourse-cdn.com/v4/letter/m/b9bd4f/32.png) [@Max\_Kossatz](https://discuss.elastic.co/u/Max_Kossatz)\ **Post date:** [April 25, 2012, 8:54am UTC](https://discuss.elastic.co/t/rabbitmq-river-bulksize/7463/1 "2012-04-25T08:54:43Z")
Hi, we are importing millions of documents a day via RabbitMQ over the RabbitMQ-river into Elasticsearch 0.19.2. Works great! The index-part of the river is configured like this: "index":{ "bulk\_size":100, "bulk\_timeout":"10s", "ordered":true } The messages we send to RabbitMQ for getting indexed are again Bulkinserts with up to 100 inserts in one message, the size varies from 1 to 100 inserts. The question is: _Would it be better to put only one insert per message for RabbitMQ, so Elasticsearch always has the same amount of data to index or does this make no difference?_ I am asking this because on the search-side we get quite very different amounts of time it take to do the same search, it varies from milliseconds to up to 20 seconds and i thought this has maybe to do with the different bulksize that Elasticsearch has to index, sometime it takes longer, sometimes not. The refresh-time for elasticsearch is set to 10s (changed from the default 1s because we experienced much lower cpu load with a higher amount). Thank you for your help, Max ---
**Author:** ![benny\_sadeh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benny_sadeh/32/2797_2.png) [@benny\_sadeh](https://discuss.elastic.co/u/benny_sadeh)\ **Post date:** [April 26, 2012, 4:06am UTC](https://discuss.elastic.co/t/rabbitmq-river-bulksize/7463/2 "2012-04-26T04:06:59Z")
my understanding is, and the rabbitmq river source code confirms, that the river is batching the requests it pulls from the queue. that bulk\_size, to the best of my knowledge, can not be changed dynamically. but maybe someone more knowledgeable can chime in ... On Apr 25, 1:54 am, Max Kossatz [max.koss...@gmail.com](mailto:max.koss...@gmail.com) wrote: > Hi, > > we are importing millions of documents a day via RabbitMQ over the > RabbitMQ-river into Elasticsearch 0.19.2. Works great! > The index-part of the river is configured like this: > "index":{ > "bulk\_size":100, > "bulk\_timeout":"10s", > "ordered":true > } > > The messages we send to RabbitMQ for getting indexed are again Bulkinserts > with up to 100 inserts in one message, the size varies from 1 to 100 > inserts. > The question is: > _Would it be better to put only one insert per message for RabbitMQ, so > Elasticsearch always has the same amount of data to index or does this make > no difference?_ > > I am asking this because on the search-side we get quite very different > amounts of time it take to do the same search, it varies from milliseconds > to up to 20 seconds and i thought this has maybe to do with the different > bulksize that Elasticsearch has to index, sometime it takes longer, > sometimes not. > > The refresh-time for elasticsearch is set to 10s (changed from the default > 1s because we experienced much lower cpu load with a higher amount). > > Thank you for your help, > Max ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [April 27, 2012, 10:12am UTC](https://discuss.elastic.co/t/rabbitmq-river-bulksize/7463/3 "2012-04-27T10:12:49Z")
The way that the logic of bulk size works is that the first message is read and all the bulk items there are added to the bulk request (regardless of the bulk size). If the bulk size is not yet reached, it will try and read (without a timeout) more messages to try and reach the bulk size, once reached, or there are no more messages, the bulk is executed. On Thu, Apr 26, 2012 at 7:06 AM, benny.sadeh [benny.sadeh@gmail.com](mailto:benny.sadeh@gmail.com) wrote: > my understanding is, and the rabbitmq river source code confirms, that > the river is batching the requests it pulls from the queue. > > that bulk\_size, to the best of my knowledge, can not be changed > dynamically. > > but maybe someone more knowledgeable can chime in ... > > On Apr 25, 1:54 am, Max Kossatz [max.koss...@gmail.com](mailto:max.koss...@gmail.com) wrote: > > > Hi, > > > > we are importing millions of documents a day via RabbitMQ over the > > RabbitMQ-river into Elasticsearch 0.19.2. Works great! > > The index-part of the river is configured like this: > > "index":{ > > "bulk\_size":100, > > "bulk\_timeout":"10s", > > "ordered":true > > } > > > > The messages we send to RabbitMQ for getting indexed are again > > Bulkinserts > > with up to 100 inserts in one message, the size varies from 1 to 100 > > inserts. > > The question is: > > _Would it be better to put only one insert per message for RabbitMQ, so > > Elasticsearch always has the same amount of data to index or does this > > make > > no difference?_ > > > > I am asking this because on the search-side we get quite very different > > amounts of time it take to do the same search, it varies from > > milliseconds > > to up to 20 seconds and i thought this has maybe to do with the different > > bulksize that Elasticsearch has to index, sometime it takes longer, > > sometimes not. > > > > The refresh-time for elasticsearch is set to 10s (changed from the > > default > > 1s because we experienced much lower cpu load with a higher amount). > > > > Thank you for your help, > > Max ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:30am UTC](https://discuss.elastic.co/t/rabbitmq-river-bulksize/7463/4 "2017-07-06T03:30:55Z")