# How does batch size effect performance in bulk indexing? **URL:** \ **Category:** Elasticsearch\ **Created:** [July 19, 2016, 7:33pm UTC](https://discuss.elastic.co/t/how-does-batch-size-effect-performance-in-bulk-indexing/55921 "2016-07-19T19:33:57Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![eugene\_miretsky](https://avatars.discourse-cdn.com/v4/letter/e/9fc348/32.png) [@eugene\_miretsky](https://discuss.elastic.co/u/eugene_miretsky)\ **Post date:** [July 19, 2016, 7:33pm UTC](https://discuss.elastic.co/t/how-does-batch-size-effect-performance-in-bulk-indexing/55921/1 "2016-07-19T19:33:57Z")
Some blogs suggest to tune the batch size for each batch to take ~1s. Why is that? ---
**Author:** ![polyfractal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/polyfractal/32/48162_2.png) [@polyfractal](https://discuss.elastic.co/u/polyfractal)\ **Post date:** [July 19, 2016, 7:46pm UTC](https://discuss.elastic.co/t/how-does-batch-size-effect-performance-in-bulk-indexing/55921/2 "2016-07-19T19:46:04Z")
I would disregard that advice... it's a very strange criteria and I don't really understand why someone would recommend it. The [general advice](https://www.elastic.co/guide/en/elasticsearch/guide/current/bulk.html#_how_big_is_too_big) is to find the largest bulk size that maximizes throughput, without negatively impacting memory usage / garbage collections. For some clusters that may be 15mb per bulk, other's may be 100mb. It really depends. But basing it on a time latency criteria isn't a good idea. ---
**Author:** ![eugene\_miretsky](https://avatars.discourse-cdn.com/v4/letter/e/9fc348/32.png) [@eugene\_miretsky](https://discuss.elastic.co/u/eugene_miretsky)\ **Post date:** [July 19, 2016, 7:51pm UTC](https://discuss.elastic.co/t/how-does-batch-size-effect-performance-in-bulk-indexing/55921/3 "2016-07-19T19:51:39Z")
Thanks! In our case, when we go from 15k to 25K docs, throughput decreases by 3x, cpu load (4 cores) goes from 360 to 270, and old Gen GC times goes up by 100x. We thought that increasing the index\_buffer\_size from 10% to 30% would resolve that, but it didn't. Any idea what is causing this behavior? Trying to understand how ES works under the good more than anything. ---
**Author:** ![polyfractal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/polyfractal/32/48162_2.png) [@polyfractal](https://discuss.elastic.co/u/polyfractal)\ **Post date:** [July 19, 2016, 7:57pm UTC](https://discuss.elastic.co/t/how-does-batch-size-effect-performance-in-bulk-indexing/55921/4 "2016-07-19T19:57:03Z")
Yep, it's the old-gen GC that's killing your performance. The bulk needs to sit in memory while it is being split/sent to the various nodes and shards. If you size the bulk too big, it just plops into memory and fills the newgen, which tenures a bunch of stuff prematurely to old-gen, which can trigger old-gen GCs. You'll notice that I specified \_physical\_sizes, not number of documents. Going by number of docs is very unreliable... 100 five-byte documents is very different from 100 ten-megabyte documents! You should really be batching based on physical size to find the optimum size. Increasing the index\_buffer\_size won't help, as this pressure is coming from the bulk object itself (not from the Lucene indexing process). Edit: Note, if you have parallel bulk threads running, you need to take that into account too. One bulk of 100mb is different from ten bulks of 100mb 🙂 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:34pm UTC](https://discuss.elastic.co/t/how-does-batch-size-effect-performance-in-bulk-indexing/55921/5 "2017-07-05T22:34:19Z")