# Bulk Insert Throughput Issues **URL:** \ **Category:** Elasticsearch\ **Created:** [September 16, 2013, 6:26pm UTC](https://discuss.elastic.co/t/bulk-insert-throughput-issues/13627 "2013-09-16T18:26:42Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Fin\_Yeates](https://avatars.discourse-cdn.com/v4/letter/f/2acd7d/32.png) [@Fin\_Yeates](https://discuss.elastic.co/u/Fin_Yeates)\ **Post date:** [September 16, 2013, 6:26pm UTC](https://discuss.elastic.co/t/bulk-insert-throughput-issues/13627/1 "2013-09-16T18:26:42Z")
I have a question regarding the throughput of the \_bulk inserting of elasticsearch, running version 0.90.3 on ubuntu 13.04 64bit. I've done a great deal of searching and have been hard pressed to find anything that appropriately covers our needs. Currently In our system we're looking to Index data at very high volumes, currently we're looking at indexing, albeit an odd parent / child relationship mapping at [https://gist.github.com/fidyeates/18cec7116926516bc033](https://gist.github.com/fidyeates/18cec7116926516bc033), under our current traffic we're looking at around indexing 1,500 'details' documents per second which equates to, on average, 12,000 documents per second needed to be indexed, at around 2-3kb size on average per document, roughly giving us a write rate of 30-40mb / second into elasticsearch, Our elasticsearch deployment is currently 2 m1.large amazon EC2 instances, configured with default configurations, 10 shards and 0 replicas. We are also rotating keys daily and flushing data thats older then two weeks. However, this does seem to be the capacity of this elasticsearch deployment, i.e. we're waiting for long amounts of time on all the \_bulk calls (tested on bulk amounts from 500-10,000 documents), and we're currently feeding in from 3 parallel threads with the calls evenly distributed across the es nodes. 1.) Are there any good performance tweaks we can make to the cluster to increase the amount of indexing throughput? I can understand if we have hit the capacity of two m1.larges (2 core, 4ECU's, 7.5gb ram). 2.) This is all disregarding search performance, as we will also be wanting to make queries on this data, can we expect to disrupt the \_bulk performance significantly by running queries on 'old' indexes? 3.) Just any tips or a point in the right direction regarding documentation (the elasticsearch docs are great though!) about this specific area would be much obliged! Cheers, Fin -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![drewr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/drewr/32/7803_2.png) [@drewr](https://discuss.elastic.co/u/drewr)\ **Post date:** [September 18, 2013, 3:35am UTC](https://discuss.elastic.co/t/bulk-insert-throughput-issues/13627/2 "2013-09-18T03:35:50Z")
Fin Yeates wrote: > 1.) Are there any good performance tweaks we can make to the > cluster to increase the amount of indexing throughput? I can > understand if we have hit the capacity of two m1.larges (2 core, > 4ECU's, 7.5gb ram). 30-40M/s is all I would expect from two m1.larges. They average 15-20M a piece. You could try indexing from multiple threads across multiple _nodes_ and see if you can increase the overall rate but I would expect those two nodes are near network capacity. > 2.) This is all disregarding search performance, as we will also > be wanting to make queries on this data, can we expect to > disrupt the \_bulk performance significantly by running queries > on 'old' indexes? Depends on what kind of data you want to pull out. I suspect you're not really taxing CPU, and, depending on what storage you use, may not even be taxing the disk. > 3.) Just any tips or a point in the right direction regarding > documentation (the elasticsearch docs are great though!) about > this specific area would be much obliged! In general ES is not the bottleneck when indexing. If you can parallelize your IO, as well as streamline whatever process feeds ES (test the rate you can send to /dev/null, for example), you can scale very well. Drew -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:15am UTC](https://discuss.elastic.co/t/bulk-insert-throughput-issues/13627/3 "2017-07-06T02:15:50Z")