I tried to find a relevant document but I could not get the answer.
Let's say there are primary shard and replica shard. I believe if the primary shard goes down, the replica shard promoted to the primary shard and recreate the replica. On the other hand, if replica shard goes down, it will simply recreate the replica based off from the primary.
The question that I have is:
- What happens when primary and replica shards go down? Do it impact other shards as well?
- Is there a way that can fully recover when both primary and replica shards go down? If there is a recovery action, what happens when there are new inserts or updates to the documents that belong to these shards during the recovery? I believe we can schedule a daily snapshot of the database but this means it will lose any new data that are coming to these shards after the snapshot was taken.
In practical even though the chances of both primary and replica shard failure are rare I believe we should still take more than one replica and distribute to different data center if possible. However, in case of the worst scenario, I am really curious how we can recover from all the primary and replica shards failure.
Any inputs are really appreciated. Thanks.