# Optimizing ES settings for bulkinserts **URL:** \ **Category:** Elasticsearch\ **Created:** [January 4, 2011, 8:31pm UTC](https://discuss.elastic.co/t/optimizing-es-settings-for-bulkinserts/3740 "2011-01-04T20:31:56Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Anurag\_Phadke](https://avatars.discourse-cdn.com/v4/letter/a/5f9b8f/32.png) [@Anurag\_Phadke](https://discuss.elastic.co/u/Anurag_Phadke)\ **Post date:** [January 4, 2011, 8:31pm UTC](https://discuss.elastic.co/t/optimizing-es-settings-for-bulkinserts/3740/1 "2011-01-04T20:31:56Z")
We are trying to insert raw-build log files inside ES. The scenario involves transferring 20 files every 10 minutes via Flume and using ES'es REST API for bulk inserts Size of single log-line: 10kb Avg. # of log-lines/second: 10K Avg. size of ES REST API call for bulk-inserts: 4mb Currently, it takes about 2 minutes for every REST API to return a HTTP 200 reply. Other than adding more hardware, are there any config settings to tweak around to speed up this process? Any idea how logstash ([http://code.google.com/p/logstash/](http://code.google.com/p/logstash/)) might be doing this? -anurag ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [January 5, 2011, 12:05pm UTC](https://discuss.elastic.co/t/optimizing-es-settings-for-bulkinserts/3740/2 "2011-01-05T12:05:32Z")
How do you index the files? Is it a single bulk request executed every 10 minutes with all the data? Would also be interesting to see where time is spent, since most http libs out there are not amazing... On Tue, Jan 4, 2011 at 10:31 PM, Anurag [anurag.phadke@gmail.com](mailto:anurag.phadke@gmail.com) wrote: > We are trying to insert raw-build log files inside ES. The scenario > involves transferring 20 files every 10 minutes via Flume and using > ES'es REST API for bulk inserts > Size of single log-line: 10kb > Avg. # of log-lines/second: 10K > Avg. size of ES REST API call for bulk-inserts: 4mb > > Currently, it takes about 2 minutes for every REST API to return a > HTTP 200 reply. Other than adding more hardware, are there any config > settings to tweak around to speed up this process? Any idea how > logstash ([Google Code Archive - Long-term storage for Google Code Project Hosting.](http://code.google.com/p/logstash/)) might be doing this? > > -anurag ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [January 5, 2011, 12:14pm UTC](https://discuss.elastic.co/t/optimizing-es-settings-for-bulkinserts/3740/3 "2011-01-05T12:14:02Z")
I added [Bulk API: Add how long the bulk API took (in milliseconds) to the response · Issue #599 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/issue/599which) includes how long the bulk execution took within ES. This will help in the future to analyze if there is something "in the middle" that takes longer. On Wed, Jan 5, 2011 at 2:05 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > How do you index the files? Is it a single bulk request executed every 10 > minutes with all the data? Would also be interesting to see where time is > spent, since most http libs out there are not amazing... > > On Tue, Jan 4, 2011 at 10:31 PM, Anurag [anurag.phadke@gmail.com](mailto:anurag.phadke@gmail.com) wrote: > > > We are trying to insert raw-build log files inside ES. The scenario > > involves transferring 20 files every 10 minutes via Flume and using > > ES'es REST API for bulk inserts > > Size of single log-line: 10kb > > Avg. # of log-lines/second: 10K > > Avg. size of ES REST API call for bulk-inserts: 4mb > > > > Currently, it takes about 2 minutes for every REST API to return a > > HTTP 200 reply. Other than adding more hardware, are there any config > > settings to tweak around to speed up this process? Any idea how > > logstash ([Google Code Archive - Long-term storage for Google Code Project Hosting.](http://code.google.com/p/logstash/)) might be doing this? > > > > -anurag ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:14am UTC](https://discuss.elastic.co/t/optimizing-es-settings-for-bulkinserts/3740/4 "2017-07-06T04:14:19Z")
Closed on July 6, 2017, 4:14am UTC