It Depends™ - if you set cluster.routing.allocation.enable: primaries as per the manual then there normally won’t be many (any?) shard movements while the node is down so recoveries will be quick, just copying any missing operations into otherwise fully-in-sync shard copies.
If you don’t set this setting then removing the node will likely trigger some rebalancing operations and then adding the node back in again will trigger more rebalancing, so it can take a while to settle.