We have a use case that requires us to replace clusters regularly. My current approach is a Blue/Green-type deployment like the following:
Cluster A is running,
bring up Cluster B and wait for it to be live.
Cluster A and B both have the same repository in S3, so snapshot Cluster A, then restore that snapshot in Cluster B.
Update the DNS to point from Cluster A to Cluster B
Delete Cluster A.
This works, but the problem is that data can be populating Cluster A while Cluster B's snapshot is being restored, so that when the DNS is updated, we lose that data.
My question is, is there a query that i can run on each index that will find all the missing documents in Cluster B? Is there another way to make sure the clusters are identical?
P.S. I know that we can update the cluster by updating each node and migrating the shards away, but lets assume that this approach is off the table for now.