I want to transfer updated data between two clusters on a daily basis (Periodically). The process fetches modified data from first cluster and update the second (if the document exists and its modified time stamp is older) . If the document Id does not exist in the second cluster (document does not exist), it should be inserted. The same thing should be done from second cluster to the first as well as both data can be modified on both with in a given time period.
- I tried to implement the task for fetching data from one cluster and update/insert into the other block by block using scroll api. does Scroll Api rereads a document which is updated during ongoing request period ?
- How can I do conditional upsert - check if document exists and if its modified time stamp is older update it or if the document does not exist index it?