# Batch submission? **URL:** \ **Category:** Elasticsearch\ **Created:** [April 16, 2010, 2:55pm UTC](https://discuss.elastic.co/t/batch-submission/2913 "2010-04-16T14:55:06Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![colinsurprenant](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/colinsurprenant/32/14776_2.png) [@colinsurprenant](https://discuss.elastic.co/u/colinsurprenant)\ **Post date:** [April 16, 2010, 2:55pm UTC](https://discuss.elastic.co/t/batch-submission/2913/1 "2010-04-16T14:55:06Z")
Hi, In the Riak mailing list [http://lists.basho.com/pipermail/riak-users\_lists.basho.com/2010-April/000927.html](http://lists.basho.com/pipermail/riak-users_lists.basho.com/2010-April/000927.html) , Eric Gaumer made the excellent suggestion of adding a batch submission endpoint in ES to avoid the HTTP overhead when dealing with very large amount of documents to submit. Is this something that could be easily added in ES? What do you think? Thanks, Colin ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [April 16, 2010, 3:38pm UTC](https://discuss.elastic.co/t/batch-submission/2913/2 "2010-04-16T15:38:49Z")
Batch submission can be added, but first, note that batch submission will not be a transactional one (either all succeed or fail). Also,instead of using batch submission, you can either multithread or async each actual operation you want to do. You should get very similar result as batching when you do it. cheers, shay.banon On Fri, Apr 16, 2010 at 5:55 PM, Colin Surprenant \< [colin.surprenant@gmail.com](mailto:colin.surprenant@gmail.com)\> wrote: > Hi, > > In the Riak mailing list > > [http://lists.basho.com/pipermail/riak-users\_lists.basho.com/2010-April/000927.html](http://lists.basho.com/pipermail/riak-users_lists.basho.com/2010-April/000927.html) > , Eric Gaumer made the excellent suggestion of adding a batch > submission endpoint in ES to avoid the HTTP overhead when dealing with > very large amount of documents to submit. > > Is this something that could be easily added in ES? What do you think? > > Thanks, > Colin ---
**Author:** ![egaumer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/egaumer/32/2365_2.png) [@egaumer](https://discuss.elastic.co/u/egaumer)\ **Post date:** [April 16, 2010, 4:21pm UTC](https://discuss.elastic.co/t/batch-submission/2913/3 "2010-04-16T16:21:45Z")
On Fri, Apr 16, 2010 at 11:38 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > Batch submission can be added, but first, note that batch submission will > not be a transactional one (either all succeed or fail). Also,instead of > using batch submission, you can either multithread or async each actual > operation you want to do. You should get very similar result as batching > when you do it. In a multithread situation (async or other wise), you still have to deal with message passing (network latency etc.) semantics. I would argue that passing batches of 1000 documents in a single thread would still be faster than spawning 1000 threads that all submit a single document. Am I wrong? Maybe at small batch sizes they are pretty equal but what about as the batch size increases? I guess I'm mainly focused on the HTTP interface and the overhead associated with this type of messaging. Batching seems like a reasonable way to reduce latency in this particular area but could very well create bottlenecks elsewhere (i.e., index writing). Even still, if multithreading is an option, then wouldn't sending batches across each of those threads be more efficient than sending one document at a time? So assume I have 100 million documents of 3K each and I _need_ to use HTTP. I plan on using 20 threads per node using 3 node feeding cluster (BTW, this is, without a doubt, a common scenario in enterprise search deployments). Being able to send a batch of a few hundred documents across each connection is going to save me a lot of HTTP calls. No? I think the transaction semantics are reasonable. If I send a batch of 200 documents, I would expect the batch to fail or succeed as one unit otherwise it's much harder for me to resubmit. This is generally how some of the commercial vendors do it. Regards, -Eric ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [April 16, 2010, 4:32pm UTC](https://discuss.elastic.co/t/batch-submission/2913/4 "2010-04-16T16:32:03Z")
Batching will certainly increase the number of documents you can index. If you use http, with keep alive, the overhead of sending one document at a time should not be that high. But, of course, it depends on a lot of factors. In Java the HTTP aspect does not add a lot of overhead (the header and such) compared to the latency of the rest of the request if you do it right, but I am not sure how much overhead you have for HTTP in ruby and others... I will add batching, and people can play with it and see if they can get better performance. Regarding the all will fail or not. I was saying that elasticsearch will _not_ support this. If you do batching, the request will hit several shards and elasticsearch will not do two phase commit across potentially many resources (shards), especially since, by itself, two phase commit is broken (but thats a different story) when it comes to many resources. The API will simply return a status for each element in the batch, i.e., if it worked or not. cheers, shay.banon On Fri, Apr 16, 2010 at 7:21 PM, Eric Gaumer [egaumer@gmail.com](mailto:egaumer@gmail.com) wrote: > On Fri, Apr 16, 2010 at 11:38 AM, Shay Banon \<[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) > > > wrote: > > > Batch submission can be added, but first, note that batch submission will > > not be a transactional one (either all succeed or fail). Also,instead of > > using batch submission, you can either multithread or async each actual > > operation you want to do. You should get very similar result as batching > > when you do it. > > In a multithread situation (async or other wise), you still have to deal > with message passing (network latency etc.) semantics. I would argue that > passing batches of 1000 documents in a single thread would still be faster > than spawning 1000 threads that all submit a single document. Am I wrong? > Maybe at small batch sizes they are pretty equal but what about as the batch > size increases? > > I guess I'm mainly focused on the HTTP interface and the overhead > associated with this type of messaging. Batching seems like a reasonable way > to reduce latency in this particular area but could very well create > bottlenecks elsewhere (i.e., index writing). > > Even still, if multithreading is an option, then wouldn't sending batches > across each of those threads be more efficient than sending one document at > a time? > > So assume I have 100 million documents of 3K each and I _need_ to use HTTP. > I plan on using 20 threads per node using 3 node feeding cluster (BTW, this > is, without a doubt, a common scenario in enterprise search deployments). > > Being able to send a batch of a few hundred documents across each > connection is going to save me a lot of HTTP calls. No? > > I think the transaction semantics are reasonable. If I send a batch of 200 > documents, I would expect the batch to fail or succeed as one unit otherwise > it's much harder for me to resubmit. This is generally how some of the > commercial vendors do it. > > Regards, > -Eric ---
**Author:** ![egaumer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/egaumer/32/2365_2.png) [@egaumer](https://discuss.elastic.co/u/egaumer)\ **Post date:** [April 16, 2010, 4:40pm UTC](https://discuss.elastic.co/t/batch-submission/2913/5 "2010-04-16T16:40:58Z")
On Fri, Apr 16, 2010 at 12:32 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: Regarding the all will fail or not. I was saying that elasticsearch will > _not_ support this. If you do batching, the request will hit several shards > and elasticsearch will not do two phase commit across potentially many > resources (shards), especially since, by itself, two phase commit is broken > (but thats a different story) when it comes to many resources. The API will > simply return a status for each element in the batch, i.e., if it worked or > not. Ahh... got ya. I think as long as people understand the limitations it's (not ideal) but okay. I think this functionality would used mainly to bootstrap an index with some pre-existing data. Once that process is complete, you'd typically switch to an incremental of near real-time feed anyway (at least that's what I'd suggest). -Eric ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [April 16, 2010, 4:53pm UTC](https://discuss.elastic.co/t/batch-submission/2913/6 "2010-04-16T16:53:47Z")
Sounds great!. Want to open an issue for the batch thingy? cheers, shay.banon On Fri, Apr 16, 2010 at 7:40 PM, Eric Gaumer [egaumer@gmail.com](mailto:egaumer@gmail.com) wrote: > On Fri, Apr 16, 2010 at 12:32 PM, Shay Banon \<[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) > > > wrote: > > Regarding the all will fail or not. I was saying that elasticsearch will > > > _not_ support this. If you do batching, the request will hit several shards > > and elasticsearch will not do two phase commit across potentially many > > resources (shards), especially since, by itself, two phase commit is broken > > (but thats a different story) when it comes to many resources. The API will > > simply return a status for each element in the batch, i.e., if it worked or > > not. > > Ahh... got ya. I think as long as people understand the limitations it's > (not ideal) but okay. I think this functionality would used mainly to > bootstrap an index with some pre-existing data. Once that process is > complete, you'd typically switch to an incremental of near real-time feed > anyway (at least that's what I'd suggest). > > -Eric ---
**Author:** ![egaumer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/egaumer/32/2365_2.png) [@egaumer](https://discuss.elastic.co/u/egaumer)\ **Post date:** [April 16, 2010, 6:04pm UTC](https://discuss.elastic.co/t/batch-submission/2913/7 "2010-04-16T18:04:06Z")
On Fri, Apr 16, 2010 at 12:53 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > Sounds great!. Want to open an issue for the batch thingy? > **[Issues · elastic/elasticsearch](https://github.com/elastic/elasticsearch/issues#issue/138)** > > Free and Open, Distributed, RESTful Search Engine. Contribute to elastic/elasticsearch development by creating an account on GitHub. -Eric ---
**Author:** ![colinsurprenant](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/colinsurprenant/32/14776_2.png) [@colinsurprenant](https://discuss.elastic.co/u/colinsurprenant)\ **Post date:** [April 20, 2010, 1:35pm UTC](https://discuss.elastic.co/t/batch-submission/2913/8 "2010-04-20T13:35:32Z")
Very nice. Thanks. Will definitely run some performance tests when its available. I agree with Eric that the typical use-case for this would be bootstrap an index with some pre-existing data. This is how I plan to use it. Having to parse the results to check the status for each element works for me. Colin On Apr 16, 2:04 pm, Eric Gaumer [egau...@gmail.com](mailto:egau...@gmail.com) wrote: > On Fri, Apr 16, 2010 at 12:53 PM, Shay Banon > [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com)wrote: > > > Sounds great!. Want to open an issue for the batch thingy? > > [Issues · elastic/elasticsearch · GitHub](http://github.com/elasticsearch/elasticsearch/issues#issue/138) > > -Eric ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:24am UTC](https://discuss.elastic.co/t/batch-submission/2913/9 "2017-07-06T04:24:29Z")