Thanks for the reply @Bernt_Rostad and your recommendation on shard sizes. I wrote this in my original post, though:
I realize this is severely oversharded, but I am trying to understand how the different shards affect the recovery process.
I was wondering why primaries recover nearly instantly, but replica's seem to be regenerated from scratch instead of being recovered. That seems to me either a bug or a very inefficient way since the data should already be there on the node. Having an arbitrary high shard count just made this issue more apparent.
About the replicas: I must've used the wrong terminology. I have a replication factor of 3 by setting number_of_replicas to 2. So on a 3 nodes cluster every index has 1 primary and 2 replicas = 3 copies of each shard.