# Deleting s3 gateway data **URL:** \ **Category:** Elasticsearch\ **Created:** [May 8, 2012, 6:59pm UTC](https://discuss.elastic.co/t/deleting-s3-gateway-data/7603 "2012-05-08T18:59:05Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Steve\_2](https://avatars.discourse-cdn.com/v4/letter/s/c77e96/32.png) [@Steve\_2](https://discuss.elastic.co/u/Steve_2)\ **Post date:** [May 8, 2012, 6:59pm UTC](https://discuss.elastic.co/t/deleting-s3-gateway-data/7603/1 "2012-05-08T18:59:05Z")
Hi, Apologies if this has been answered before; I've have a look at the docs and archives but may have missed something. We've got a single index on 0.18.6 running on an EC2 cluster that we want to back with the s3 gateway. We generate a few GB of documents a day at a regular pace (no huge spikes), and set a TTL of 15 days on all documents. As such, we're hoping that the size of the data stored in s3 will remain reasonably constant over time. Two questions: 1 - is that much data too much to hope to push to s3 all the time? 2 - will elasticsearch remove old data from s3 as it becomes unneeded? Thanks! Steve ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [May 10, 2012, 6:02am UTC](https://discuss.elastic.co/t/deleting-s3-gateway-data/7603/2 "2012-05-10T06:02:54Z")
First, my general recommendation is to use local gateway on ec2, its considerably more lightweight compared to the s3 gateway. You can still back it up ofcourse. On Tue, May 8, 2012 at 9:59 PM, Steve [steve1.mclellan@googlemail.com](mailto:steve1.mclellan@googlemail.com)wrote: > Hi, > > Apologies if this has been answered before; I've have a look at the > docs and archives but may have missed something. We've got a single > index on 0.18.6 running on an EC2 cluster that we want to back with > the s3 gateway. We generate a few GB of documents a day at a regular > pace (no huge spikes), and set a TTL of 15 days on all documents. As > such, we're hoping that the size of the data stored in s3 will remain > reasonably constant over time. Two questions: > > 1 - is that much data too much to hope to push to s3 all the time? It should be fine. > 2 - will elasticsearch remove old data from s3 as it becomes unneeded? Not as data gets deleted. Deletes are marked to be deleted, and later on, as the index performs merges, they will be merged out. > Thanks! > > Steve ---
**Author:** ![Steve\_2](https://avatars.discourse-cdn.com/v4/letter/s/c77e96/32.png) [@Steve\_2](https://discuss.elastic.co/u/Steve_2)\ **Post date:** [May 11, 2012, 3:21pm UTC](https://discuss.elastic.co/t/deleting-s3-gateway-data/7603/3 "2012-05-11T15:21:00Z")
Hi Shay, Thanks for the response. Our motivation to use the s3 gateway is that for some of our environments we have a lot of documents on relatively small clusters (because load is low) and were finding ourselves firing up more ec2 instances just to get more local disk space, which isn't very cost effective. We're trialling the s3 gateway for a couple of weeks to see how it goes; we may investigate other options if we have problems. Steve On May 10, 1:02 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > First, my general recommendation is to use local gateway on ec2, its > considerably more lightweight compared to the s3 gateway. You can still > back it up ofcourse. > > On Tue, May 8, 2012 at 9:59 PM, Steve [steve1.mclel...@googlemail.com](mailto:steve1.mclel...@googlemail.com)wrote: > > > Hi, > > > Apologies if this has been answered before; I've have a look at the > > docs and archives but may have missed something. We've got a single > > index on 0.18.6 running on an EC2 cluster that we want to back with > > the s3 gateway. We generate a few GB of documents a day at a regular > > pace (no huge spikes), and set a TTL of 15 days on all documents. As > > such, we're hoping that the size of the data stored in s3 will remain > > reasonably constant over time. Two questions: > > > 1 - is that much data too much to hope to push to s3 all the time? > > It should be fine. > > > 2 - will elasticsearch remove old data from s3 as it becomes unneeded? > > Not as data gets deleted. Deletes are marked to be deleted, and later on, > as the index performs merges, they will be merged out. > > > Thanks! > > > Steve ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [May 15, 2012, 7:54pm UTC](https://discuss.elastic.co/t/deleting-s3-gateway-data/7603/4 "2012-05-15T19:54:46Z")
With the s3 gateway, you still have the indexes stored on the nodes data location, they are just snapshotted periodically to s3. On Fri, May 11, 2012 at 6:21 PM, Steve [steve1.mclellan@googlemail.com](mailto:steve1.mclellan@googlemail.com)wrote: > Hi Shay, > > Thanks for the response. Our motivation to use the s3 gateway is that > for some of our environments we have a lot of documents on relatively > small clusters (because load is low) and were finding ourselves firing > up more ec2 instances just to get more local disk space, which isn't > very cost effective. We're trialling the s3 gateway for a couple of > weeks to see how it goes; we may investigate other options if we have > problems. > > Steve > > On May 10, 1:02 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > First, my general recommendation is to use local gateway on ec2, its > > considerably more lightweight compared to the s3 gateway. You can still > > back it up ofcourse. > > > > On Tue, May 8, 2012 at 9:59 PM, Steve \<[steve1.mclel...@googlemail.com](mailto:steve1.mclel...@googlemail.com) > > wrote: > > > > > Hi, > > > > > Apologies if this has been answered before; I've have a look at the > > > docs and archives but may have missed something. We've got a single > > > index on 0.18.6 running on an EC2 cluster that we want to back with > > > the s3 gateway. We generate a few GB of documents a day at a regular > > > pace (no huge spikes), and set a TTL of 15 days on all documents. As > > > such, we're hoping that the size of the data stored in s3 will remain > > > reasonably constant over time. Two questions: > > > > > 1 - is that much data too much to hope to push to s3 all the time? > > > > It should be fine. > > > > > 2 - will elasticsearch remove old data from s3 as it becomes unneeded? > > > > Not as data gets deleted. Deletes are marked to be deleted, and later on, > > as the index performs merges, they will be merged out. > > > > > Thanks! > > > > > Steve ---
**Author:** ![Eric\_Jain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eric_jain/32/834_2.png) [@Eric\_Jain](https://discuss.elastic.co/u/Eric_Jain)\ **Post date:** [May 15, 2012, 8:57pm UTC](https://discuss.elastic.co/t/deleting-s3-gateway-data/7603/5 "2012-05-15T20:57:19Z")
On Tue, May 15, 2012 at 12:54 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > With the s3 gateway, you still have the indexes stored on the nodes data > location, they are just snapshotted periodically to s3. Is this snapshotting less efficient than other backup options? ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [May 16, 2012, 9:14pm UTC](https://discuss.elastic.co/t/deleting-s3-gateway-data/7603/6 "2012-05-16T21:14:39Z")
Yes, because the recovery on full cluster restart relies on hte latest snapshotted data, while if it was based on the local gateway by default, and only on backups on worst case scenario, then its a different story. On Tue, May 15, 2012 at 11:57 PM, Eric Jain [eric.jain@gmail.com](mailto:eric.jain@gmail.com) wrote: > On Tue, May 15, 2012 at 12:54 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > With the s3 gateway, you still have the indexes stored on the nodes data > > location, they are just snapshotted periodically to s3. > > Is this snapshotting less efficient than other backup options? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:28am UTC](https://discuss.elastic.co/t/deleting-s3-gateway-data/7603/7 "2017-07-06T03:28:30Z")