Snapshots - restoring an index from a large repository with long term storage (LTO ...)

Elastic Stack Elasticsearch
1

I have a classic question about snapshots in elasticsearch.

I make elasticsearch snapshots to some repository - for example, let it be some Network File System (NFS) or similar.

Later this snapshot from this network storage goes to other media - for example, tapes LTO or BD-ROM.

The question is simple:

There are 10 indexes in the repository.

What if I only want to restore 1 index?

Do I have to load the entire repository into NFS from the tape (some long term medium) and then recover only one index from elasticsearch?

Is there any idea that I need to retrieve only one index from the LTO/BD-ROM?

Limiting yourself to just the index you need can save you a lot of time.

2

Yes.

3

Well....

In real systems, what are the models for creating repositories?

Create fewer large repositories or more but smaller repositories?

Repository size here can be a big role.

Are there any effective approaches when you are not sure what you are must recovery in future?

4

Today there are a handful of recognised problems with using offline/long-term storage for backups:

  • There isn't even a procedure for correctly copying a registered repository to tape.

  • As you discovered here, there isn't really a procedure for selectively restoring certain indices from offline storage (e.g. AWS Glacier, but the fix for tape would be similar IMO).

  • Version N cannot restore snapshots from versions before version N-1 (I don't even see a Github issue for this).

The fact that we don't get very many comments or +1s on those issues indicates it's not something that a lot of folks are trying to do. There are workarounds, although they're not ideal:

  • Unregister the repository each time you copy it to tape.

  • Use more repositories, each containing a subset of indices.

  • Reindex your whole archive at each major version upgrade.

Please feel free to indicate your need for these features on the issues I linked above.

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.