As I mentioned earlier as well, our system is ok with eventual consistency but we cannot afford downtime. That is why I am currently using a two pass approach which is as follows:
Phase 0: Pre Migration checks
pre_migration_check(dest_client, source_client, source_index)
Checking whether cluster2 has enough disk space to store source_index or not.
Phase 1: Change write pipeline
change_pipeline(source_client, pipeline_name, new_index, source_index)
Setting _ingest pipeline for source_index to field: _index, value: new_index, so that all the updates and writes from now on for source_index go to new_index. Doing this to segregate writes from this point in time.
Phase 2: Snapshot and restore [First Pass]
snaps(source_client, source_repository, source_index, dest_client, dest_index)
Taking snapshot and restoring it to the dest_index.
Phase 2.1: Verifying snapshot and restore
verifying_snapshot_and_restore(source_client, source_index, dest_client, dest_index)
Verifying by comparing the count of documents in source_index and dest_index.
Making pipeline null in destination index
make_null(dest_client , dest_index)
Setting default pipeline to null in dest_index as it has also been restored from source_index.
---- Shift operations from source_index to dest_index from client side.----
Phase 3: Reindexing new index [2nd Pass]
reindex_data(source_host, dest_client, new_index, dest_index)
Reindexing the new_index having the delta documents that got added while the first snapshot and restore was running, so that the dest_index becomes up to date.
Phase 3.1: Post second pass verification
second_pass_verification(source_client, source_index, new_index, dest_client, dest_index)
Verifying by comparing the count of documents in source_index + new_index and dest_index.
Phase 4: Cleanup
delete_index(source_client , source_index)
delete_index(source_client , new_index)
Deleting or downgrading indices in cluster1 as now they are irrelevant.
Obviously there will be some redundant documents in the process because this whole process cannot incorporate deletes on the main source_index. Let me know if you see any other flaws in this.