# Reindex API: parallel reindex requests, reindexing while still indexing to source indices **URL:** \ **Category:** Elasticsearch\ **Created:** [July 7, 2017, 8:45pm UTC](https://discuss.elastic.co/t/reindex-api-parallel-reindex-requests-reindexing-while-still-indexing-to-source-indices/92331 "2017-07-07T20:45:30Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![mpereira](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mpereira/32/15804_2.png) [@mpereira](https://discuss.elastic.co/u/mpereira)\ **Post date:** [July 7, 2017, 8:45pm UTC](https://discuss.elastic.co/t/reindex-api-parallel-reindex-requests-reindexing-while-still-indexing-to-source-indices/92331/1 "2017-07-07T20:45:30Z")
We upgraded ES from 2 to 5 and want to do a full reindex so that our indices are upgradable to ES6. Given clusters running on [AWS D2 nodes](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/storage-optimized-instances.html) each with hundreds of +100GB indices configured to have 1 primary and 0 replicas, what would be an optimal reindexing strategy? Nodes don't have a lot of extra disk space (~80% full). What we're currently thinking is reindexing/deleting 1 index at a time with `wait_for_completion=true` but initial tests show that this takes a long time. We're seeing average throughput of 4.5MBps. Would it make sense to drop `wait_for_completion=true` and let the cluster parallelize reindex tasks? Would the cluster retry reindexes that failed due to a temporary lack of disk space? Does it parallelize `wait_for_completion=false` reindex requests without specifying [slicing](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html#docs-reindex-automatic-slice)? What happens when reindexing from indices which are still being written to? Would the destination index only get documents that were in the source at the point in time when the reindex request was sent? ---
**Author:** ![mpereira](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mpereira/32/15804_2.png) [@mpereira](https://discuss.elastic.co/u/mpereira)\ **Post date:** [July 10, 2017, 3:53pm UTC](https://discuss.elastic.co/t/reindex-api-parallel-reindex-requests-reindexing-while-still-indexing-to-source-indices/92331/2 "2017-07-10T15:53:47Z")
Would you expect a reindex on a single-node cluster to have an average throughput of 4MB/s given these node IO metrics? ```auto $ mount | grep elasticsearch /dev/mapper/ephemeral-elasticsearch on /mnt/elasticsearch type xfs (rw) ``` ```auto $ df -h | grep elasticsearch /dev/mapper/ephemeral-elasticsearch 5.4T 4.8T 678G 88% /mnt/elasticsearch ``` ```auto $ sudo hdparm -Tt /dev/mapper/ephemeral-elasticsearch /dev/mapper/ephemeral-elasticsearch: Timing cached reads: 18780 MB in 2.00 seconds = 9400.56 MB/sec Timing buffered disk reads: 920 MB in 3.01 seconds = 306.07 MB/sec ``` ```auto $ sudo dd if=/dev/zero of=/mnt/elasticsearch/output bs=8k count=100k; sudo rm -f /mnt/elasticsearch/output 102400+0 records in 102400+0 records out 838860800 bytes (839 MB) copied, 2.17104 s, 386 MB/s ``` ---
**Author:** ![mpereira](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mpereira/32/15804_2.png) [@mpereira](https://discuss.elastic.co/u/mpereira)\ **Post date:** [July 10, 2017, 7:47pm UTC](https://discuss.elastic.co/t/reindex-api-parallel-reindex-requests-reindexing-while-still-indexing-to-source-indices/92331/3 "2017-07-10T19:47:47Z")
> [@mpereira](#): > > Would the destination index only get documents that were in the source at the point in time when the reindex request was sent? Answer to this seems to be yes after experiments. ---
**Author:** ![mpereira](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mpereira/32/15804_2.png) [@mpereira](https://discuss.elastic.co/u/mpereira)\ **Post date:** [July 12, 2017, 5:29pm UTC](https://discuss.elastic.co/t/reindex-api-parallel-reindex-requests-reindexing-while-still-indexing-to-source-indices/92331/4 "2017-07-12T17:29:48Z")
> [@mpereira](#): > > Would you expect a reindex on a single-node cluster to have an average throughput of 4MB/s given these node IO metrics? > > $ mount | grep elasticsearch > /dev/mapper/ephemeral-elasticsearch on /mnt/elasticsearch type xfs (rw) > > $ df -h | grep elasticsearch > /dev/mapper/ephemeral-elasticsearch 5.4T 4.8T 678G 88% /mnt/elasticsearch > > $ sudo hdparm -Tt /dev/mapper/ephemeral-elasticsearch > /dev/mapper/ephemeral-elasticsearch: > Timing cached reads: 18780 MB in 2.00 seconds = 9400.56 MB/sec > Timing buffered disk reads: 920 MB in 3.01 seconds = 306.07 MB/sec > > $ sudo dd if=/dev/zero of=/mnt/elasticsearch/output bs=8k count=100k; sudo rm -f /mnt/elasticsearch/output > 102400+0 records in > 102400+0 records out > 838860800 bytes (839 MB) copied, 2.17104 s, 386 MB/s I hate to bump this, but could someone at least opine on whether this throughput rate is normal? Do you need more information? ---
**Author:** ![dakrone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dakrone/32/23351_2.png) [@dakrone](https://discuss.elastic.co/u/dakrone)\ **Post date:** [August 4, 2017, 12:50am UTC](https://discuss.elastic.co/t/reindex-api-parallel-reindex-requests-reindexing-while-still-indexing-to-source-indices/92331/5 "2017-08-04T00:50:15Z")
@nik9000 this sounds like something you might be able to help with? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 1, 2017, 12:50am UTC](https://discuss.elastic.co/t/reindex-api-parallel-reindex-requests-reindexing-while-still-indexing-to-source-indices/92331/6 "2017-09-01T00:50:35Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.