# Bad performance with varying bulk size **URL:** \ **Category:** Elasticsearch\ **Created:** [October 17, 2016, 7:21pm UTC](https://discuss.elastic.co/t/bad-performance-with-varying-bulk-size/63222 "2016-10-17T19:21:15Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![mcrandy](https://avatars.discourse-cdn.com/v4/letter/m/54ee81/32.png) [@mcrandy](https://discuss.elastic.co/u/mcrandy)\ **Post date:** [October 17, 2016, 7:21pm UTC](https://discuss.elastic.co/t/bad-performance-with-varying-bulk-size/63222/1 "2016-10-17T19:21:15Z")
Hi! We're currently having some indexing performance issues with one of our applications using elasticsearch 1.7. We're writing to one index (16 shards, 21 data nodes) with 10 concurrent threads using the Java API. Bulk sizes depends on size of incoming data and since incoming data varies in size so do the bulk sizes. As long as the bulks sizes meets our preferred max size of 8000 documents performance is great for all threads. However, as soon as incoming data load decreases for some threads and bulk size strives towards 1000 documents or less the import performance are degraded for all threads. Even for threads that have a bulk sizes of 8000. Are there some configuration parameters one should think of with the described scenario with varying bulksize to maximise performance or do we need to even out the bulk sizes to have a stable indexing rate? /Alex ---
**Author:** ![mcrandy](https://avatars.discourse-cdn.com/v4/letter/m/54ee81/32.png) [@mcrandy](https://discuss.elastic.co/u/mcrandy)\ **Post date:** [October 17, 2016, 8:06pm UTC](https://discuss.elastic.co/t/bad-performance-with-varying-bulk-size/63222/2 "2016-10-17T20:06:23Z")
Some more information. - Heap for our data nodes are 32GB - Our indices are immutable so we toggle refresh rate. -1 before indexing starts and 1 when indexing completes. - Replicas are 0 until we're done writing to an index. - We're using spinning disks so we have set index.merge.scheduler.max\_thread\_count to 1. - index.translog.flush\_threshold\_size is set to 1000mb - indices.memory.index\_buffer\_size: 50% - indices.store.throttle.type: none (despite that we see some throttling going on every now and then) ---
**Author:** ![mcrandy](https://avatars.discourse-cdn.com/v4/letter/m/54ee81/32.png) [@mcrandy](https://discuss.elastic.co/u/mcrandy)\ **Post date:** [October 21, 2016, 4:20pm UTC](https://discuss.elastic.co/t/bad-performance-with-varying-bulk-size/63222/3 "2016-10-21T16:20:50Z")
We managed to get the speed up by sorting incoming data first, making the bulks more even. That have increased the import speed but it's not stable. It's toggling between very fast (200k-300k docs/sec) to super low (1k-10k doc/s) in short period of time (minutes). When it's slow we can see that some batches get "stuck" in the "bulk.active" queue for a data node. It's different data node pretty much every time so it doesn't seem to be related to a single data node or machine. We can't see that our system resources are saturated in any way during this and there seems to be no throttling going on (at least according to the logs). ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [October 21, 2016, 4:35pm UTC](https://discuss.elastic.co/t/bad-performance-with-varying-bulk-size/63222/4 "2016-10-21T16:35:31Z")
Do you supply doc ids or just let elasticsearch generate new ones? If you supply the IDs there is extra work in checking if the doc already exists - it's fast but essentially introduces a read step to every write. ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [October 21, 2016, 4:55pm UTC](https://discuss.elastic.co/t/bad-performance-with-varying-bulk-size/63222/5 "2016-10-21T16:55:41Z")
> [@mcrandy](#): > > That have increased the import speed but it's not stable. It's toggling between very fast (200k-300k docs/sec) to super low (1k-10k doc/s) in short period of time (minutes). That's pretty normal, if you look at the volume throughput (megabytes per second) you should see a constant rate. Docs per second are pretty meaningless when doc size varies much. Bulk requests can be sized by volume to send uniformly sized requests, so I suggest to examine the bulk API. ---
**Author:** ![mcrandy](https://avatars.discourse-cdn.com/v4/letter/m/54ee81/32.png) [@mcrandy](https://discuss.elastic.co/u/mcrandy)\ **Post date:** [October 21, 2016, 5:11pm UTC](https://discuss.elastic.co/t/bad-performance-with-varying-bulk-size/63222/6 "2016-10-21T17:11:04Z")
We do. We generate a doc id using file name (with path) together with the row number from the file the document was taken from. The row number is zero padded. As you can imagine we're getting quite long id's. We can see a lot of disk reads indeed but not enough to be a problem. ---
**Author:** ![mcrandy](https://avatars.discourse-cdn.com/v4/letter/m/54ee81/32.png) [@mcrandy](https://discuss.elastic.co/u/mcrandy)\ **Post date:** [October 21, 2016, 5:11pm UTC](https://discuss.elastic.co/t/bad-performance-with-varying-bulk-size/63222/7 "2016-10-21T17:11:42Z")
We're using the BulkRequestBuilder right now but been considering the BulkProcessor .. ---
**Author:** ![mcrandy](https://avatars.discourse-cdn.com/v4/letter/m/54ee81/32.png) [@mcrandy](https://discuss.elastic.co/u/mcrandy)\ **Post date:** [October 21, 2016, 5:23pm UTC](https://discuss.elastic.co/t/bad-performance-with-varying-bulk-size/63222/8 "2016-10-21T17:23:38Z")
Regarding "index.merge.scheduler.max\_thread\_count". We're using spinning disks and have set count to 1 according to the recommendations. However, we got a lot of them and they're pretty much the fatest ones that you come by. Rest of the hardware is also quite beefy. The 7 machines hosting 3 data nodes each got CPUs with 24 cores. Are we still obligated to just have 2+1 merge threads? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:10pm UTC](https://discuss.elastic.co/t/bad-performance-with-varying-bulk-size/63222/9 "2017-07-05T22:10:27Z")