Why is reindex from remote constantly slowing down on large indices?

Hey,
I have 2 clusters:
Source cluster: 5.6 version, 3 node with 126 GB RAM on each.
Dest cluster: 7.9 version, 3 node with 6 GB RAM on each and ES_JAVA_OPTS -Xms3g -Xmx3g

I have ~60 indices of different sizes from (100items/100kb) to (30million items/16Gb) biggest and all of this I want to move between clusters. Total amount of pri.store.size is ~26Gb.
I perform steps from https://www.elastic.co/guide/en/elasticsearch/reference/current/reindex-upgrade-remote.html and set refresh_interval=-1 and number_of_replicas=0 to indices but I see that on indices with some significant size (>~1Gb) reindex task constantly slowing down.
With 3 tasks at time (same as node number) it took around 1-2 days to transfer 26Gb of data.

I created script which tracks number of transferred docs, I collect this data every 10 seconds from task info on dest cluster. And this is how it looks on plot:
Transfer progress (based on created items / target item number) (don't look at plot jump from 10 to 11 percentages, I just lost this data)

Number of transferred docs for last 10 seconds

I suppose that there is no networking issue. We can imagine that all of this happens in different US AWS datacenters.

I tried to set source.size to 10k, 3k since all my docs are small. But there was no significant difference.

In dest cluster logs I see nothing strange except two following things:
Error:

Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)

and messages like:

[gc][5841] overhead, spent [491ms] collecting in the last [1.4s]

My main question: What I'm doing wrong and how I can speedup reindex from remote?

Thanks.
Looking forward any advices.

UPD
As result after ~5-6 hours of data transferring total speed may down to 60 docs per second. ~24kb/s
:frowning:

See this possibly related issue which contains information on the version with a fix and a workaround for earlier versions.

2 Likes

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.