Hi, In our production environment, we take backup of our ES daily to S3 repository. Now since the process is year long and we have a huge list of snapshot, we decided to move the old snapshots from S3 to Glacier. Following are my questions related to the process: 1.) I presume moving snapshots fro…

In addition to the previous questions, we have noticed that the snapshot which are moved from S3 to glacier using Amazon life cycle configurations and are not deleted permanently from glacier are visible in S3 console and their storage class is marked as Glacier. When the ES tries to create the new …

Glacier is an AWS abstraction so I don't know if ES will be able to tell? I'll pass it on though and see if it's anything we can support.

I wonder if the problem (in part) is the response times of Glacier? From Secure Storage - Amazon S3 Glacier storage classes - AWS To keep costs low, Amazon Glacier is optimized for infrequently accessed data where a retrieval time of several hours is suitable.

I put all of this into here https://github.com/elastic/elasticsearch/issues/12500

Thanks @warkolm . :blush: . Also can you help me on the questions I had in the very first post on how snapshot thingy works in case of a deletion of snapshot? How does other snapshot preserves the state of the backup data. Thanks again. Piyush

Cause it works on a shard level, it'll hold onto any shards it needs to be able to restore any remaining snapshots and just delete the shards it doesn't need.

Does that mean deletion of any older snapshot actually would not cause loosing of data if we try to restore back from recent snapshots and restoring back from a new snapshot will provide the complete data set? The incremental word mentioned in the documentation is causing a lot of confusion.

It works on the segment level within a shard, and each segment is only copied once even if it is still in use for multiple snapshots. If you remove old snapshots, segments will only be deleted once there are no more snapshots linking to them.

You beat me to it :stuck_out_tongue:

lol..!! :smiley: another question for sake of my clarity to both of you: So just like ES works with segments where merging and all those processes happen at back end, the same happens for snapshots as well? The first snapshot copies the segment and as the new snapshots are created, the new segments …

Snapshot and restore | S3 and Glacier

Elastic Stack Elasticsearch

warkolm (Mark Walkom) July 28, 2015, 6:25am 14

If you delete old snapshots ES is smart enough to only delete the segments it won't need to be able to restore every other snapshot.

Your second comment is correct.

Topic		Replies	Views
Managing Snapshot Files from Outside ES Elasticsearch	3	492	July 6, 2017
Is it possible to move a one time snapshot to Glacier? Elasticsearch snapshot-and-restore	5	1180	April 7, 2023
Snapshot management on S3 Elasticsearch slm-snapshot-lifecycle-management	2	473	June 28, 2020
Delete S3 snapshots through API to minimize the size of s3 bucket Elasticsearch	15	1650	November 12, 2019
What is the best practice for periodic snapshotting with awc-cloud+s3 Elasticsearch	8	2405	July 6, 2017

Snapshot and restore | S3 and Glacier

Related topics