We're currently utilizing Elasticsearch version 8.8.1 and storing our daily snapshots in S3 using the 'Intelligent_Tiering' storage class. In an effort to optimize costs, based on AWS's documentation on Intelligent Tiering says that " objects not accessed for 180 days are moved to the Deep Archive …

[image] mihai1: data won't be instantly accessible I haven't tried it, but I'm pretty sure this means it won't work with Elasticsearch snapshots today. Elasticsearch needs the data to be instantly accessible.

thank you @DavidTurner for your answer. However, if we manually initiate the RestoreObject process, the data will automatically transition to the Frequent Access Tier and become accessible again. Elasticsearch should then be able to detect it. right? On the other side, I think one potential drawba…

That's true, but I don't think there's a way for you to work out which objects need to be restored.

One suggestion, but I'm not sure this will work for you, is to use another cloud provider, in this case Google Cloud Storage. We had similar requirements to save money with snapshots and decided to use GCP because how their storage tiering works. When you move between the storage classes, it will …

GCS's autoclass feature is pretty similar to S3's Intelligent Tiering, it's just that S3 goes even colder than GCS does with their Deep-Archive tier. But that's an opt-in feature, it should work fine on S3 as long as you don't opt in. (NB I haven't done a pricing comparison between the two, maybe G…

S3 Intelligent-Tiering class with Deep Archive Access tier for snapshots is working?

Elastic Stack Elasticsearch

DavidTurner (David Turner) September 4, 2023, 3:52pm 3

Amazon S3 Intelligent-Tiering Storage Class | AWS says:

If the object you are retrieving is stored in the optional Deep Archive tier, before you can retrieve the object you must first restore a copy using RestoreObject.

Elasticsearch definitely doesn't know how to do this.

Topic		Replies	Views
Intelligent Tiering & Snapshots Elasticsearch	4	841	March 13, 2019
S3 Snapshots (Storage Tiers) Elasticsearch	1	302	October 10, 2024
Elasticsearch backup - S3 repository Elasticsearch	2	225	April 28, 2023
Snapshot policy will continue to backup fail when the backup jumps out of shard error Kibana snapshot-and-restore	7	426	August 22, 2023
Is it possible to move a one time snapshot to Glacier? Elasticsearch snapshot-and-restore	5	1179	April 7, 2023

S3 Intelligent-Tiering class with Deep Archive Access tier for snapshots is working?

Related topics