How to understand snapshot repositories on s3?

Elastic Stack Elasticsearch
1

Hello,

We have our ES snapshot stored on S3 currently via repository-s3 plugin. Per this post:

I'd like to copy repository data of certain indices to local disk.

I see each snapshot has an UUID ( /_snapshot/REPOSITORY/my_snapshot/_status), but in my S3 bucket, I cannot use this UUID to locate the index.

And I cannot find any elastic doc talking about this.

Any help is appreciated!

LH

PS: in the index-###.dms file in the REPOSITORY bucket on s3, I also see some snapshots has a "state:1" attribute while not others. But /_status reports all are "SUCCESS". Any thought?

index-111.dms

{"snapshots":[{"name":"snapshot_1508396416","uuid":"lGcXoXU7QDeE9MWyPjd3vA","state":1},{"name":"snapshot_20171001","uuid":"CHvjCqBqQ7S6waMpymAyeA"}

2

I'd like to copy repository data of certain indices to local disk.

Why do you want to do this?

3

mainly to avoid waiting for downloading from s3.

I figure this way I can use aws cli to sync/download a particular snapshot to my local fs, then restore from it at much faster speed. Otherwise, every time I need to wait for the data to be downloaded on demand from s3.

The other dumb choice is then to just sync the whole repository bucket to local disk, configure the repository-fs plugin pointing to local dir, then restore the particular snapshot using the _snapshot api.

4

I see.

Then yes, copying the whole S3 repo locally is probably what you could do.

Note that you can also:

  • Restore from S3
  • Backup to local FS

Then you will just have to restore from Local FS again.

That's better IMO as you will use only elasticsearch API instead of having to copy "files" manually.

5

Got it. Thanks for confirming David!

OTOH, won't disclose how the repository directory is populated on S3? :slight_smile:

6

I don't know from the top of my head.
And the source code is open! :smiley:

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.