Hey,
I have 2 clusters:
Source cluster: 5.6 version, 3 node with 126 GB RAM on each.
Dest cluster: 7.9 version, 3 node with 6 GB RAM on each and ES_JAVA_OPTS -Xms3g -Xmx3g
I have ~60 indices of different sizes from (100items/100kb) to (30million items/16Gb) biggest and all of this I want to move between clusters. Total amount of pri.store.size is ~26Gb.
I perform steps from https://www.elastic.co/guide/en/elasticsearch/reference/current/reindex-upgrade-remote.html and set refresh_interval=-1 and number_of_replicas=0 to indices but I see that on indices with some significant size (>~1Gb) reindex task constantly slowing down.
With 3 tasks at time (same as node number) it took around 1-2 days to transfer 26Gb of data.
I created script which tracks number of transferred docs, I collect this data every 10 seconds from task info on dest cluster. And this is how it looks on plot:
Transfer progress (based on created items / target item number) (don't look at plot jump from 10 to 11 percentages, I just lost this data)
Number of transferred docs for last 10 seconds
I suppose that there is no networking issue. We can imagine that all of this happens in different US AWS datacenters.
I tried to set source.size to 10k, 3k since all my docs are small. But there was no significant difference.
In dest cluster logs I see nothing strange except two following things:
Error:
Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)
and messages like:
[gc][5841] overhead, spent [491ms] collecting in the last [1.4s]
My main question: What I'm doing wrong and how I can speedup reindex from remote?
Thanks.
Looking forward any advices.
UPD
As result after ~5-6 hours of data transferring total speed may down to 60 docs per second. ~24kb/s
