Snapshot & Restore clarification

[r.ganeshbabu (ganeshbabu)]

Hi all,

We are using Elasticsearch 1.7.2 in Prod and having 3 master nodes, 3 data nodes & 1 client node. ES cluster has 3 indices such as,

Index name Primary size Shards
es_chr, 2 GB 1
es_cval 173 GB 6
es_item 850 GB 18

Yesterday we took snapshot in Production cluster and I can see in repository snapshot files/folders were generated with based on the number of shards allotted and then we restored the indices in DR(Di aster Recovery) cluster for backup purpose. Restoring in DR cluster, I manually moved the snapshot files from PROD to DR server (through SCP) and I started restoring it and process were completed well. Suppose today If i am going to take snapshot of index es_item, data might be changed or new data will be added in index.

1. Should I again manually need to move snapshot files from PROD to DR server for backup?
   (or)

2. Is there any other workaround to move incremental data?

3. How to find the difference between initial & incremental snapshot?

Please kindly help me in this..

Thanks,
Ganeshbabu R

[dadoonet (David Pilato)]

If I get it correctly, I'd rsync the "repository" in PROD DC to DR DC.

I would not try to copy "only" a snapshot. Snapshot files are shared within the same repository.

[r.ganeshbabu (ganeshbabu)]

Thanks for your response @dadoonet

We don't have a common repository for PROD & DR server. For PROD server we created NAS mount for repository to take backup but for the DR server location were different so that we can't rsync the repository to DR cluster. So we created local repository in DR server and manually moving snapshot files from PROD repository to DR local repository.

Please give your feedback to overcome this problem.

Thanks,
Ganeshbabu R

[dadoonet (David Pilato)]

I don't know. I'm not a sys guy.

But you need to add all files available in the PROD repository to the DR repository.
I don't know how you can "diff" the two directories... May be sorting by date and only copy the "new" files?

[r.ganeshbabu (ganeshbabu)]

Thanks for your response @dadoonet

[shubham9799 (Shubham Agrawal)]

I took second snapshot after adding some files and try to find the difference. I found that some files also got removed and some got modified. But if I am adding documents, why some files get removed from the backup?

[Christian_Dahlqvist]

Please start a new thread for your question.

[shubham9799 (Shubham Agrawal)]

Files change during snapshot and restore process

I took second snapshot after adding some files and try to find the difference. I found that some files also got removed and some got modified. But if I am adding documents, why some files get removed from the backup? I am asking this because I want to copy the new files to a different location. If some files are getting modified or deleted, then I have no option left and copy the whole repository folder which will be very time consuming.

Here is the new thread as asked.