# Why is reindex from remote constantly slowing down on large indices? **URL:** \ **Category:** Elasticsearch\ **Tags:** reindex\ **Created:** [December 2, 2020, 12:27pm UTC](https://discuss.elastic.co/t/why-is-reindex-from-remote-constantly-slowing-down-on-large-indices/257354 "2020-12-02T12:27:13Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Dmitry\_O](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dmitry_o/32/80095_2.png) [@Dmitry\_O](https://discuss.elastic.co/u/Dmitry_O)\ **Post date:** [December 2, 2020, 12:27pm UTC](https://discuss.elastic.co/t/why-is-reindex-from-remote-constantly-slowing-down-on-large-indices/257354/1 "2020-12-02T12:27:13Z")
Hey, I have 2 clusters: Source cluster: 5.6 version, 3 node with 126 GB RAM on each. Dest cluster: 7.9 version, 3 node with 6 GB RAM on each and `ES_JAVA_OPTS -Xms3g -Xmx3g` I have ~60 indices of different sizes from (100items/100kb) to (30million items/16Gb) biggest and all of this I want to move between clusters. Total amount of pri.store.size is ~26Gb. I perform steps from [https://www.elastic.co/guide/en/elasticsearch/reference/current/reindex-upgrade-remote.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/reindex-upgrade-remote.html) and set `refresh_interval=-1 and number_of_replicas=0` to indices but I see that on indices with some significant size (\>~1Gb) reindex task constantly slowing down. With 3 tasks at time (same as node number) it took around 1-2 days to transfer 26Gb of data. I created script which tracks number of transferred docs, I collect this data every 10 seconds from task info on dest cluster. And this is how it looks on plot: Transfer progress (based on created items / target item number) (don't look at plot jump from 10 to 11 percentages, I just lost this data) ![Screen Shot 2020-12-02 at 2.57.13 PM](https://us1.discourse-cdn.com/elastic/original/3X/e/2/e2b9dbf2039de1a731bd7efa7bc86a3b34c73a7e.png) Number of transferred docs for last 10 seconds ![Screen Shot 2020-12-02 at 2.57.18 PM](https://us1.discourse-cdn.com/elastic/original/3X/3/d/3deb4807d7846735b7f9a7428f3a0dc25e9ee7d6.jpeg) I suppose that there is no networking issue. We can imagine that all of this happens in different US AWS datacenters. I tried to set source.size to 10k, 3k since all my docs are small. But there was no significant difference. In dest cluster logs I see nothing strange except two following things: Error: ```auto Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16) ``` and messages like: ```auto [gc][5841] overhead, spent [491ms] collecting in the last [1.4s] ``` **My main question: What I'm doing wrong and how I can speedup reindex from remote?** Thanks. Looking forward any advices. UPD As result after ~5-6 hours of data transferring total speed may down to 60 docs per second. ~24kb/s ☹ ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [December 3, 2020, 9:47am UTC](https://discuss.elastic.co/t/why-is-reindex-from-remote-constantly-slowing-down-on-large-indices/257354/2 "2020-12-03T09:47:54Z")
See this possibly [related issue](https://github.com/elastic/elasticsearch/issues/65780) which contains information on the version with a fix and a workaround for earlier versions. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 31, 2020, 9:47am UTC](https://discuss.elastic.co/t/why-is-reindex-from-remote-constantly-slowing-down-on-large-indices/257354/3 "2020-12-31T09:47:55Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.