# Why doesn't the Reindex API parallelize by shard automatically? **URL:** \ **Category:** Elasticsearch\ **Created:** [September 14, 2016, 5:14pm UTC](https://discuss.elastic.co/t/why-doesnt-the-reindex-api-parallelize-by-shard-automatically/60532 "2016-09-14T17:14:19Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![aaron\_ximm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron_ximm/32/61229_2.png) [@aaron\_ximm](https://discuss.elastic.co/u/aaron_ximm)\ **Post date:** [September 14, 2016, 5:14pm UTC](https://discuss.elastic.co/t/why-doesnt-the-reindex-api-parallelize-by-shard-automatically/60532/1 "2016-09-14T17:14:19Z")
As has been reported elsewhere here, we too were very surprised at how slow the Reindex API is if you simply follow the documentation, and issue a straight reindex. For reference we have a 9M doc/4 TB index, simple clean schema but large documents, on a well provisioned 10-data-node [+masters/clients] 2.3.4 cluster under minimal load... and it is taking several days to reindex, peaking at at best 100 doc/s and averaging more like 25. This is with an identical schema and no source filtering or rewriting etc... just a simple pour-over from one shard count to a higher one. Reading here I found the suggestion to perform manual parallelization by effectively 'sharding' the index according to some feature of the source. Presumably filtering against a unique key. I'd like to request that in its stable form (5.x+) the Reindex API at minimum automatically perform concurrent scroll/bulk posts by shard. Presumably the same logic used to shard documents by id could also be used to partition into parallel read processes. It would be invaluable to automatically make each shard reindex in parallel. (The ideal of course would be that there's simply an option in the initiating operation, say `"concurrency": 10` ... It's funny that there are dials for throttling, but the common experience seems to be that an over-exuberant reindexing process is the least of anyone's worries!) We can dream...! Aaron ---
**Author:** ![kstaken](https://avatars.discourse-cdn.com/v4/letter/k/e68b1a/32.png) [@kstaken](https://discuss.elastic.co/u/kstaken)\ **Post date:** [September 14, 2016, 6:30pm UTC](https://discuss.elastic.co/t/why-doesnt-the-reindex-api-parallelize-by-shard-automatically/60532/2 "2016-09-14T18:30:51Z")
This won't help you with the built in API but we built a tool called Teraslice because we needed much higher throughput for these types of operations along with the ability to cross clusters. Recently used it to migrate 71B records from a single cluster into 8 clusters in less than 14 hours. Something like 1.2M docs/s. > **[terascope/teraslice](https://github.com/terascope/teraslice)** > > teraslice - Distributed computing platform for processing JSON data stored in Elasticsearch Kimbro ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 14, 2016, 10:05pm UTC](https://discuss.elastic.co/t/why-doesnt-the-reindex-api-parallelize-by-shard-automatically/60532/3 "2016-09-14T22:05:58Z")
This is a known limitation, we're working on improving it for 5.0. ---
**Author:** ![aaron\_ximm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron_ximm/32/61229_2.png) [@aaron\_ximm](https://discuss.elastic.co/u/aaron_ximm)\ **Post date:** [September 20, 2016, 12:14am UTC](https://discuss.elastic.co/t/why-doesnt-the-reindex-api-parallelize-by-shard-automatically/60532/4 "2016-09-20T00:14:46Z")
Awesome news, thanks Mark. Speed notwithstanding the API has been a godsend. We had opportunity to put the 'create' operation filter to test when the node doing the index was rebooted before the process finished, and the fact that our 'resume' of reindexing required only adding a parameter to the original post was lovely. We're going to experiment with an application-level parallelization this week, also; we have a motivating test case that requires some additional fields in our index which have to be fetched per document. Best, Aaron ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [September 20, 2016, 12:35am UTC](https://discuss.elastic.co/t/why-doesnt-the-reindex-api-parallelize-by-shard-automatically/60532/5 "2016-09-20T00:35:07Z")
Fetching things per document sounds like a thing that should be handled by an external script rather than reindex. It'll probably work though.... This thread did remind me that I needed to work on parallelizing reindex. I reached out to the author of parallel scroll and we talked through a plan. I'll put together a proposal soon-ish, though it isn't going to make it for 5.0. 5.1 if all goes well. ---
**Author:** ![aaron\_ximm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron_ximm/32/61229_2.png) [@aaron\_ximm](https://discuss.elastic.co/u/aaron_ximm)\ **Post date:** [September 20, 2016, 5:03pm UTC](https://discuss.elastic.co/t/why-doesnt-the-reindex-api-parallelize-by-shard-automatically/60532/6 "2016-09-20T17:03:39Z")
Hi Nik, Indeed, for this run we're going to do an application-side reindex! I did briefly consider whether there was value in maybe first indexing the fields we need to add into a temporary secondary index, then doing document merges (with all the data actually be fetchable as existing ES documents in the same cluster), but that's definitely overkill for our immediate case. 🙂 Thanks for the update! Aaron ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [October 5, 2016, 6:42pm UTC](https://discuss.elastic.co/t/why-doesnt-the-reindex-api-parallelize-by-shard-automatically/60532/7 "2016-10-05T18:42:38Z")
> [@nik9000](#): > > This thread did remind me that I needed to work on parallelizing reindex. I > reached out to the author of parallel scroll and we talked through a plan. > I'll put together a proposal soon-ish, though it isn't going to make it for > 5.0. 5.1 if all goes well. I did finally open this: > > > Adds support for \`?slices=N\` to reindex which automatically > parallelizes the pr…ocess using parallel scrolls on \`\_uid\`. Performance > testing sees a 3x performance improvement for simple docs > on decent hardware, maybe 30% performance improvement > for more complex docs. Still compelling, especially because > clusters should be able to get closer to the 3x than the 30% > number. > > Closes #20624 > > > > Edit: this used to say the below things but I've since changed it to make it reflect what I ended up implementing: > > Adds support for \`?workers=N\` to reindex which automatically > parallelizes the process using parallel scrolls on \`\_uid\`. Simple > performance testing sees a 3x performance improvement. At this point I targeted it at 5.1. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:14pm UTC](https://discuss.elastic.co/t/why-doesnt-the-reindex-api-parallelize-by-shard-automatically/60532/8 "2017-07-05T22:14:38Z")