# Sweet spot for bulk indexing **URL:** \ **Category:** Elasticsearch\ **Created:** [January 31, 2013, 9:44pm UTC](https://discuss.elastic.co/t/sweet-spot-for-bulk-indexing/10577 "2013-01-31T21:44:41Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Robin\_Verlangen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robin_verlangen/32/1542_2.png) [@Robin\_Verlangen](https://discuss.elastic.co/u/Robin_Verlangen)\ **Post date:** [January 31, 2013, 9:44pm UTC](https://discuss.elastic.co/t/sweet-spot-for-bulk-indexing/10577/1 "2013-01-31T21:44:41Z")
Hi there, For my use case I will be bulk indexing 24/7. Average size of a document is somewhere between 500-1000 bytes. Is there a certain sweet spot for bulk indexing? Currently I use a bucket size of 10 events. I could imagine I should go for a benchmark, but maybe there's someone out here with a better supported opinion. Best regards, Robin Verlangen _Software engineer_ \* \* W [http://www.robinverlangen.nl](http://www.robinverlangen.nl) E robin@us2.nl [http://goo.gl/Lt7BC](http://goo.gl/Lt7BC) Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [February 1, 2013, 12:32am UTC](https://discuss.elastic.co/t/sweet-spot-for-bulk-indexing/10577/2 "2013-02-01T00:32:31Z")
Do you use the Java bulk indexing or HTTP bulk indexing? It's hard to give info just for a known document size. Very much depends on your hardware, and even more on software, most important multithreading. Have you complex analyzers? How fast can your indexer build the documents? This will be CPU intensive tasks. I have now some hardware available for testing and I started some experiments with bulk indexing recently. An average desktop PC can index ~10.000 docs of the stated size using the Java API, standard settings, single node, with an indexer that can use all CPU cores. But a typical OOTB bulk indexing setup is around 1000 docs for a single thread on a current CPU/disk system. You can find the sweet spot of your system by using bulk index throttling. Define a bulk action size and a maximum number of concurrency and, if exceeded, wait for open bulk requests until they are completed. If the bulk indexing runs smoothly without delays caused by CPU or IO you're fine. Best regards, Jörg Am 31.01.13 22:44, schrieb Robin Verlangen: > Hi there, > > For my use case I will be bulk indexing 24/7. Average size of a > document is somewhere between 500-1000 bytes. Is there a certain sweet > spot for bulk indexing? Currently I use a bucket size of 10 events. > > I could imagine I should go for a benchmark, but maybe there's someone > out here with a better supported opinion. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Robin\_Verlangen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robin_verlangen/32/1542_2.png) [@Robin\_Verlangen](https://discuss.elastic.co/u/Robin_Verlangen)\ **Post date:** [February 1, 2013, 7:45am UTC](https://discuss.elastic.co/t/sweet-spot-for-bulk-indexing/10577/3 "2013-02-01T07:45:04Z")
Hi Jörg, _Do you use the Java bulk indexing or HTTP bulk indexing?_ Java bulk _It's hard to give info just for a known document size. Very much depends on your hardware, and even more on software, most important multithreading._ Currently we use "CPU\_CORES\*1" amount of indexing threads that all have a bulk size of (currently) 10. _Have you complex analyzers? How fast can your indexer build the documents? This will be CPU intensive tasks._ Uses the default tokenizer. The indexers don't seem to be a bottleneck, they are capable of sending through 1 million plus events per minute. The rest seems to make sense. I'll just go with the bench marking, that would probably the best way. I then could use my results to automatically throttle the indexing. Best regards, Robin Verlangen _Software engineer_ \* \* W [http://www.robinverlangen.nl](http://www.robinverlangen.nl) E robin@us2.nl [http://goo.gl/Lt7BC](http://goo.gl/Lt7BC) Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies. On Fri, Feb 1, 2013 at 1:32 AM, Jörg Prante [joergprante@gmail.com](mailto:joergprante@gmail.com) wrote: > Do you use the Java bulk indexing or HTTP bulk indexing? > > It's hard to give info just for a known document size. Very much depends > on your hardware, and even more on software, most important multithreading. > > Have you complex analyzers? How fast can your indexer build the documents? > This will be CPU intensive tasks. > > I have now some hardware available for testing and I started some > experiments with bulk indexing recently. An average desktop PC can index > ~10.000 docs of the stated size using the Java API, standard settings, > single node, with an indexer that can use all CPU cores. But a typical OOTB > bulk indexing setup is around 1000 docs for a single thread on a current > CPU/disk system. > > You can find the sweet spot of your system by using bulk index throttling. > Define a bulk action size and a maximum number of concurrency and, if > exceeded, wait for open bulk requests until they are completed. If the bulk > indexing runs smoothly without delays caused by CPU or IO you're fine. > > Best regards, > > Jörg > > Am 31.01.13 22:44, schrieb Robin Verlangen: > > Hi there, > > > For my use case I will be bulk indexing 24/7. Average size of a document > > is somewhere between 500-1000 bytes. Is there a certain sweet spot for bulk > > indexing? Currently I use a bucket size of 10 events. > > > > I could imagine I should go for a benchmark, but maybe there's someone > > out here with a better supported opinion. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elasticsearch+unsubscribe@\*\*[googlegroups.com](http://googlegroups.com)[elasticsearch%2Bunsubscribe@googlegroups.com](mailto:elasticsearch%2Bunsubscribe@googlegroups.com) > . > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > . -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:53am UTC](https://discuss.elastic.co/t/sweet-spot-for-bulk-indexing/10577/4 "2017-07-06T02:53:38Z")