# Elasticsearch bulk index **URL:** \ **Category:** Elasticsearch\ **Created:** [April 5, 2013, 8:53am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-index/11459 "2013-04-05T08:53:29Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Alexis\_Okuwa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alexis_okuwa/32/1983_2.png) [@Alexis\_Okuwa](https://discuss.elastic.co/u/Alexis_Okuwa)\ **Post date:** [April 5, 2013, 8:53am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-index/11459/1 "2013-04-05T08:53:29Z")
I am working on a monitoring project were the server sends each requests metric data back to elasticsearch, to be honest once per minute there shipped off to the montoring project where they needed to be inserted into elasticsearch i am already using bulk index but there could be times were there are thounands of items to be added at once i am not sure if elasticsearch can handle something like this and what steps are skipped with bulk index or if there are any better ideas. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [April 5, 2013, 10:00am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-index/11459/2 "2013-04-05T10:00:45Z")
How much volume do you transfer per bulk request? Note, there are some limits you should observe: if you use HTTP, check if you need to raise the upload limit of 100 MB ( http.max\_content\_length), and, in the ES node, check if your bulk requests fits into the heap memory. It certainly does, but if you transmit some GBs at once you might get into some trouble concerning memory usage and bandwidth (request transfer time). I recommend to organize concurrent bulk requests, to all available nodes, and do not foget to evaluate the BulkResponse callbacks to limit concurrency, so the cluster does not get overloaded. Jörg Am 05.04.13 10:53, schrieb Wojons Tech: > I am working on a monitoring project were the server sends each > requests metric data back to elasticsearch, to be honest once per > minute there shipped off to the montoring project where they needed to > be inserted into elasticsearch i am already using bulk index but there > could be times were there are thounands of items to be added at once i > am not sure if elasticsearch can handle something like this and what > steps are skipped with bulk index or if there are any better ideas. -- > You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Alexis\_Okuwa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alexis_okuwa/32/1983_2.png) [@Alexis\_Okuwa](https://discuss.elastic.co/u/Alexis_Okuwa)\ **Post date:** [April 5, 2013, 8:22pm UTC](https://discuss.elastic.co/t/elasticsearch-bulk-index/11459/3 "2013-04-05T20:22:43Z")
Okay that is pretty interesting to think about, I am trying to strucre the data to be smaller and removing redundant, information.would using syncnous or async replication be better, when having large bulk inserts. On Fri, Apr 5, 2013 at 3:00 AM, Jörg Prante [joergprante@gmail.com](mailto:joergprante@gmail.com) wrote: > How much volume do you transfer per bulk request? > > Note, there are some limits you should observe: if you use HTTP, check if > you need to raise the upload limit of 100 MB ( http.max\_content\_length), > and, in the ES node, check if your bulk requests fits into the heap memory. > It certainly does, but if you transmit some GBs at once you might get into > some trouble concerning memory usage and bandwidth (request transfer time). > > I recommend to organize concurrent bulk requests, to all available nodes, > and do not foget to evaluate the BulkResponse callbacks to limit > concurrency, so the cluster does not get overloaded. > > Jörg > > Am 05.04.13 10:53, schrieb Wojons Tech: > > > I am working on a monitoring project were the server sends each requests > > metric data back to elasticsearch, to be honest once per minute there > > shipped off to the montoring project where they needed to be inserted into > > elasticsearch i am already using bulk index but there could be times were > > there are thounands of items to be added at once i am not sure if > > elasticsearch can handle something like this and what steps are skipped > > with bulk index or if there are any better ideas. -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to elasticsearch+unsubscribe@\*\*[googlegroups.com](http://googlegroups.com)[elasticsearch%2Bunsubscribe@googlegroups.com](mailto:elasticsearch%2Bunsubscribe@googlegroups.com) > > . > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > . > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit [https://groups.google.com/d/](https://groups.google.com/d/)\*\* > topic/elasticsearch/Ry44\_\*\*CdVZn0/unsubscribe?hl=en-US[https://groups.google.com/d/topic/elasticsearch/Ry44\_CdVZn0/unsubscribe?hl=en-US](https://groups.google.com/d/topic/elasticsearch/Ry44_CdVZn0/unsubscribe?hl=en-US) > . > To unsubscribe from this group and all its topics, send an email to > elasticsearch+unsubscribe@\*\*[googlegroups.com](http://googlegroups.com)[elasticsearch%2Bunsubscribe@googlegroups.com](mailto:elasticsearch%2Bunsubscribe@googlegroups.com) > . > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > . -- Enjoy, Alexis Okuwa WojonsTech 424.835.1223 -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Jilles\_van\_Gurp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jilles_van_gurp/32/879_2.png) [@Jilles\_van\_Gurp](https://discuss.elastic.co/u/Jilles_van_Gurp)\ **Post date:** [April 5, 2013, 11:12pm UTC](https://discuss.elastic.co/t/elasticsearch-bulk-index/11459/4 "2013-04-05T23:12:34Z")
The upcoming 0.90 has a lot of performance optimizations for this and should vastly reduce the impact of doing big bulk additions on a live cluster. Your best option is probably to bulk index maybe a thousand documents at a time. You can actually use multiple threads as well to speed things up. With all the lucene 4 work in 0.90, this should have only a minimal impact. Everything has a price of course but this sounds like it should be no problem. I've seen Elasticsearch deal with bulk indexing tens of thousands of documents per second while still being able to serve read traffic. Jilles On Friday, April 5, 2013 10:53:29 AM UTC+2, Wojons Tech wrote: > I am working on a monitoring project were the server sends each requests > metric data back to elasticsearch, to be honest once per minute there > shipped off to the montoring project where they needed to be inserted into > elasticsearch i am already using bulk index but there could be times were > there are thounands of items to be added at once i am not sure if > elasticsearch can handle something like this and what steps are skipped > with bulk index or if there are any better ideas. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Alexis\_Okuwa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alexis_okuwa/32/1983_2.png) [@Alexis\_Okuwa](https://discuss.elastic.co/u/Alexis_Okuwa)\ **Post date:** [April 5, 2013, 11:50pm UTC](https://discuss.elastic.co/t/elasticsearch-bulk-index/11459/5 "2013-04-05T23:50:28Z")
okay perfect i cant wait to use it with what i am working on currently elasticsearch is doign an amazing job on my small test vm, i will post a few more questions about other features i am thinking about using. On Fri, Apr 5, 2013 at 4:12 PM, Jilles van Gurp [jillesvangurp@gmail.com](mailto:jillesvangurp@gmail.com)wrote: > The upcoming 0.90 has a lot of performance optimizations for this and > should vastly reduce the impact of doing big bulk additions on a live > cluster. Your best option is probably to bulk index maybe a thousand > documents at a time. You can actually use multiple threads as well to speed > things up. With all the lucene 4 work in 0.90, this should have only a > minimal impact. Everything has a price of course but this sounds like it > should be no problem. I've seen Elasticsearch deal with bulk indexing tens > of thousands of documents per second while still being able to serve read > traffic. > > Jilles > > On Friday, April 5, 2013 10:53:29 AM UTC+2, Wojons Tech wrote: > > > I am working on a monitoring project were the server sends each requests > > metric data back to elasticsearch, to be honest once per minute there > > shipped off to the montoring project where they needed to be inserted into > > elasticsearch i am already using bulk index but there could be times were > > there are thounands of items to be added at once i am not sure if > > elasticsearch can handle something like this and what steps are skipped > > with bulk index or if there are any better ideas. > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/Ry44\_CdVZn0/unsubscribe?hl=en-US](https://groups.google.com/d/topic/elasticsearch/Ry44_CdVZn0/unsubscribe?hl=en-US) > . > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- Enjoy, Alexis Okuwa WojonsTech 424.835.1223 -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:42am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-index/11459/6 "2017-07-06T02:42:31Z")