How to understand snapshot repositories on s3?

[lhong1]

Hello,

We have our ES snapshot stored on S3 currently via repository-s3 plugin. Per this post:

Snapshots: are type "fs" compatible with type "s3"?

Is it possible to do a snapshot type "fs" (locally), sync the local snapshot location to an s3 bucket, and then from an AWS elasticsearch instance run a restore using the s3 bucket containing the snapshot generated with type "fs"? Know I could directly sync the snapshot to "s3" with a local install of repository-s3 plugin ; however find this to be incredibly slow with little to no feedback as to what is happening. Prefer the speed of a local snapshot (local is source of truth) and then to see t…

I'd like to copy repository data of certain indices to local disk.

I see each snapshot has an UUID ( /_snapshot/REPOSITORY/my_snapshot/_status), but in my S3 bucket, I cannot use this UUID to locate the index.

And I cannot find any elastic doc talking about this.

Any help is appreciated!

LH

PS: in the index-###.dms file in the REPOSITORY bucket on s3, I also see some snapshots has a "state:1" attribute while not others. But /_status reports all are "SUCCESS". Any thought?

index-111.dms

{"snapshots":[{"name":"snapshot_1508396416","uuid":"lGcXoXU7QDeE9MWyPjd3vA","state":1},{"name":"snapshot_20171001","uuid":"CHvjCqBqQ7S6waMpymAyeA"}

[dadoonet (David Pilato)]

I'd like to copy repository data of certain indices to local disk.

Why do you want to do this?

[lhong1]

mainly to avoid waiting for downloading from s3.

I figure this way I can use aws cli to sync/download a particular snapshot to my local fs, then restore from it at much faster speed. Otherwise, every time I need to wait for the data to be downloaded on demand from s3.

The other dumb choice is then to just sync the whole repository bucket to local disk, configure the repository-fs plugin pointing to local dir, then restore the particular snapshot using the _snapshot api.

[dadoonet (David Pilato)]

I see.

Then yes, copying the whole S3 repo locally is probably what you could do.

Note that you can also:

• Restore from S3
• Backup to local FS

Then you will just have to restore from Local FS again.

That's better IMO as you will use only elasticsearch API instead of having to copy "files" manually.

[lhong1]

Got it. Thanks for confirming David!

OTOH, won't disclose how the repository directory is populated on S3?

[dadoonet (David Pilato)]

I don't know from the top of my head.
And the source code is open!