What snapshot data should I copy from one cluster to another for a migration?

Elastic Stack Elasticsearch
1

Hi, I'm trying to migrate some indices from one Elasticsearch cluster to another in a completely different datacenter. They share basically no common infrastructure. Both clusters are on version 1.6.0 and the latter cluster is currently empty.

My process on how to do this:

  1. Create a snapshot in the source cluster.
  2. Manually copy and move the data from the source cluster to the target cluster.
  3. In the target cluster create a snapshot repository configured at the location of where I copied the snapshots.
  4. Make a POST /_snapshot/.../_restore request in the target cluster to restore the data.

My main question is about what data I need to copy. Our source has several large indices (all in all a couple TBs) and many snapshots, and I'm unsure about what data I actually need to move to the target cluster.

The source cluster's underlying repository mount is structured like so. (This is a superset of the data i'm going to copy to the target cluster):

[path.repo]
├── index
├── indices
│ ├── [index 1]
│ │ ├── 0
│ │ │ ├── __0
│ │ │ | ...
│ │ │ ├── snapshot-manual-snapshot-1
│ │ │ ├── snapshot-manual-snapshot-3
│ │ ├── 1
│ │ │ ├── __0
│ │ │ ├── ...
│ │ │ ├── snapshot-manual-snapshot-1
│ │ │ ├── snapshot-manual-snapshot-3
│ │ ├── 2
│ │ │ ├── __0
│ │ │ ├── ...
│ │ │ ├── snapshot-manual-snapshot-1
│ │ │ └── snapshot-manual-snapshot-3
│ │ ├── 3
│ │ │ ├── __0
│ │ │ ├── ...
│ │ │ ├── snapshot-manual-snapshot-1
│ │ │ └── snapshot-manual-snapshot-3
│ │ ├── 4
│ │ │ ├── __0
│ │ │ ├── ...
│ │ │ ├── snapshot-manual-snapshot-1
│ │ │ └── snapshot-manual-snapshot-3
│ │ ├── snapshot-manual-snapshot-1
│ │ └── snapshot-manual-snapshot-3
│ └── [index 2]
│ ├── 0
│ │ ├── __0
│ │ ├── ...
│ │ ├── snapshot-manual-snapshot-1
│ │ ├── snapshot-manual-snapshot-2
│ │ └── snapshot-manual-snapshot-3
│ ├── 1
│ │ ├── __0
│ │ ├── ...
│ │ ├── snapshot-manual-snapshot-1
│ │ ├── snapshot-manual-snapshot-2
│ │ ├── snapshot-manual-snapshot-3
│ ├── 2
│ │ ├── __0
│ │ ├── ...
│ │ ├── snapshot-manual-snapshot-1
│ │ ├── snapshot-manual-snapshot-2
│ │ ├── snapshot-manual-snapshot-3
│ ├── 3
│ │ ├── __0
│ │ ├── ...
│ │ ├── snapshot-manual-snapshot-1
│ │ ├── snapshot-manual-snapshot-2
│ │ ├── snapshot-manual-snapshot-3
│ ├── 4
│ │ ├── __0
│ │ ├── ...
│ │ ├── snapshot-manual-snapshot-1
│ │ ├── snapshot-manual-snapshot-2
│ │ └── snapshot-manual-snapshot-3
│ ├── snapshot-manual-snapshot-1
│ ├── snapshot-manual-snapshot-2
│ └── snapshot-manual-snapshot-3
├── metadata-manual-snapshot-1
├── metadata-manual-snapshot-2
├── metadata-manual-snapshot-3
├── snapshot-manual-snapshot-1
├── snapshot-manual-snapshot-2
└── snapshot-manual-snapshot-3

If I wanted to restore only "manual-snapshot-2" that includes "index 2" what data would need to be copied to the target cluster's file system? Thanks

2

Elasticsearch 1.6 is very, very old and has been EOL a loooong time, so you should really upgrade. I have not touched an Elasticsearch 1.x cluster in years but think the snapshot and restore functionality changed relatively little for a long time.

The answer is that you will need to copy the whole repository as segments can be reused between snapshots if they have not changed in order to ensure the restore operation will work and the repository is valid.

3

Thanks for the response! Agreed about needing to upgrade. We are currently in the process of migrating our data to an up-to-date cluster, still a work in progress though.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.