Lots of shards relocating post restore

Welcome to the community.

I assume you don't want to pay the license to use CCR?

Can you confirm number_of_replicas on DC1 and DC2 (e.g. do you restore with 0 replicas, then increase it afterwards?) ? And was this issue always so, and if not when did it start ?

I ask because there was somewhere in 8.x release cycle that it seems the "balancing" algorithm was changed, and this led to more shard movements to "balance" what at first glance appeared to be already-balanced clusters. There have been a couple of threads on here about this since I've been active on the forum. Here's one.

My recollection is that tho you can try to tune a cluster to be less prone to shard movements, but it's a bit of a dark art, and the algorithm seems a bit opaque (well, the code is open-source, so that's not quite fair, but I don't think the "controls" are the clearest documented, IMO).

Did you try doing the restores exactly one-index-at-a-time, and if so does it lead to the same shuffling shard behavior?

I hope one of the other regular posters is able to provide ideas/suggestions here, as I could see how frustrating the reported behaviour must be.

btw, a bit of a wild idea if nothing else pans out, but if your DC2 cluster had just one (beefy) node, then it would not be necessary to move any shards around. OK, you'd have no replicas, but other cluster nodes could be added as part of the recovery process. You could even have 2x single-node DC2 clusters.