Thanks, Mark.
Regardless of how many shards we have, I am still trying to understand the behavior.
Because I thought the delayed_unassgined _shards will help with the recovery time since shard shuffling across the cluster won't happen.
However, we are still seeing shards being rebalanced across the cluster and the recovery time is very long.
Is it because re-allocation takes some time for every shard, so before ES can re-allocate all 3k "delayed unassigned shards" back to the original datanode, the 5 minute window (delay_time) has passed, resulting in the whole cluster to start the shard-shuffling dance still?
Thanks,
Chelsey