# Index Backups to S3?
**URL:** https://discuss.elastic.co/t/index-backups-to-s3/10832
**Category:** Elasticsearch
**Created:** [February 21, 2013, 2:39am UTC](https://discuss.elastic.co/t/index-backups-to-s3/10832 "2013-02-21T02:39:37Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@Bruno\_Miranda](https://discuss.elastic.co/u/Bruno_Miranda)
#### Post date: [February 21, 2013, 2:39am UTC](https://discuss.elastic.co/t/index-backups-to-s3/10832/1 "2013-02-21T02:39:37Z")
I have a 3 node cluster on ec2. All 3 nodes run as master eligible/data
notes. Default 1 replica and 5 shards.
I am wondering if backing up the index is necessary. If so, is S3 a good
place to put it?
Our entire index can be recreated from MySQL in about 12 hours. Can you
guys please point me in the right direction?
Thank you.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@karmi](https://discuss.elastic.co/u/karmi)
#### Post date: [February 21, 2013, 8:19am UTC](https://discuss.elastic.co/t/index-backups-to-s3/10832/2 "2013-02-21T08:19:29Z")
On EC2, I'd say the best backup option is an EBS snapshot -- if you're
using EBS for ES persistence, that is.
The recommended, general backup/restore strategy right now is to use
tar+scp/rsync/etc to offload the whole data directory somewhere else. That
somewhere could well be S3, you can script it with the Fog gem [1](http://fog.io/storage/). Maybe
you can reuse ideas or code from the Backup gem [2](https://github.com/meskyanichi/backup).
Karel
On Thursday, February 21, 2013 3:39:37 AM UTC+1, Bruno Miranda wrote:
> I have a 3 node cluster on ec2. All 3 nodes run as master eligible/data
> notes. Default 1 replica and 5 shards.
>
> I am wondering if backing up the index is necessary. If so, is S3 a good
> place to put it?
>
> Our entire index can be recreated from MySQL in about 12 hours. Can you
> guys please point me in the right direction?
>
> Thank you.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@karmi](https://discuss.elastic.co/u/karmi)
#### Post date: [February 21, 2013, 8:22am UTC](https://discuss.elastic.co/t/index-backups-to-s3/10832/3 "2013-02-21T08:22:40Z")
> Maybe you can reuse ideas or code from the Backup gem [2].
Specifically the "Amazon S3" section on the syncers page:
[Syncers · backup/backup Wiki · GitHub](https://github.com/meskyanichi/backup/wiki/Syncers) looks intriguing.
Karel
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Bruno\_Miranda](https://discuss.elastic.co/u/Bruno_Miranda)
#### Post date: [February 21, 2013, 6:11pm UTC](https://discuss.elastic.co/t/index-backups-to-s3/10832/4 "2013-02-21T18:11:11Z")
Any reason why I should not use S3 Gateway?
> **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)**
>
> Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today.
On Thursday, February 21, 2013 12:22:40 AM UTC-8, Karel Minařík wrote:
> Maybe you can reuse ideas or code from the Backup gem [2].
>
> >
>
> Specifically the "Amazon S3" section on the syncers page:
> [Syncers · backup/backup Wiki · GitHub](https://github.com/meskyanichi/backup/wiki/Syncers) looks intriguing.
>
> Karel
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@karmi](https://discuss.elastic.co/u/karmi)
#### Post date: [February 21, 2013, 6:25pm UTC](https://discuss.elastic.co/t/index-backups-to-s3/10832/5 "2013-02-21T18:25:06Z")
> Any reason why I should not use S3 Gateway?
Yes: it's deprecated and will be removed.
Karel
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Ivan](https://discuss.elastic.co/u/Ivan)
#### Post date: [February 21, 2013, 6:25pm UTC](https://discuss.elastic.co/t/index-backups-to-s3/10832/6 "2013-02-21T18:25:25Z")
S3 gateway has been deprecated:
>
>
> Shared gateways (shared FS storage or S3 for example) are problematic performanc…e wise since they constantly need to snapshot the state of the index to a shared location, and then use that as the system of record. The local gateway on the other hand doesn't need it, and performs much better.
>
> The main benefit of a shared gateway is the fact that the data is actually stored on another persistent location (i.e. using ephemeral disks on AWS, but still having the data on s3), but then its actually abusing the shared gateway design (to be used as a backup).
>
> In the near future, we will have a proper snapshot(backup)/restore API, which will be the proper way to do backups, but relaying on the shared gateway for that is problematic. Note, backups can still be made by "rsync" the data location for each node "manually".
On Thu, Feb 21, 2013 at 10:11 AM, Bruno Miranda [bru.miranda@gmail.com](mailto:bru.miranda@gmail.com)wrote:
> Any reason why I should not use S3 Gateway?
>
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/modules/gateway/s3.html)
>
> On Thursday, February 21, 2013 12:22:40 AM UTC-8, Karel Minařík wrote:
>
> > Maybe you can reuse ideas or code from the Backup gem [2].
> >
> > >
> >
> > Specifically the "Amazon S3" section on the syncers page:
> > [https://github.com/\*\*meskyanichi/backup/wiki/\*\*Syncers](https://github.com/ **meskyanichi/backup/wiki/** Syncers)[https://github.com/meskyanichi/backup/wiki/Syncers](https://github.com/meskyanichi/backup/wiki/Syncers) looks
> > intriguing.
> >
> > Karel
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Nick\_Zadrozny](https://discuss.elastic.co/u/Nick_Zadrozny)
#### Post date: [February 21, 2013, 8:01pm UTC](https://discuss.elastic.co/t/index-backups-to-s3/10832/7 "2013-02-21T20:01:32Z")
On Wed, Feb 20, 2013 at 7:39 PM, Bruno Miranda [bru.miranda@gmail.com](mailto:bru.miranda@gmail.com)wrote:
> I am wondering if backing up the index is necessary. If so, is S3 a good
> place to put it?
Here's what I would recommend. It's based on what we do for backups at
[http://bonsai.io/](http://bonsai.io/) and is an alternative to EBS snapshots, which are a
pretty reasonable approach if you're serving your data from a single EBS
volume. (I think there are arguments for _not_ using EBS; another topic.)
First, `cp -lr` your Elasticsearch data directory for a quick, cheap
filesystem snapshot. This is one of those arcane bits of Unix knowledge
that I learned once and understand intuitively, but probably would do a bad
job explaining in detail, so consult man(1). Effectively, you get a cheap,
instant copy and only pay the disk space for the delta as your original
changes.
Incidentally, `cp -lr` is cheap and useful enough that we use it to
snapshot our data on every deploy, just in case.
If your data _is_ on EBS, I would first `rsync -a` that snapshot over to
the ephemeral store. This presumes you have enough space on your ephemeral
store, which is a good constraint to consider when designing your cluster.
An up-to-date ephemeral copy gives you some fairly cheap insurance when
(not if) your EBS volume gets stuck. You can just change your data
directory and restart the cluster. It should also save you some iops
against your production EBS volume traffic while you're running your backup
to S3.
From the data snapshot, or rsync'd copy in your ephemeral store, you can
use something akin to s3sync to send your data over to S3. We wrote a
custom implementation; the backup gem that Karel linked looks reasonable
too. We're also syncing into a "rolling window" of S3 buckets per daily
backup, with a directory per host, since our main story for full backups is
recovering from a customer's own accidental deletion.
When considering availability in AWS, I have these priorities:
1. Replicate every index in at least two Availability Zones. You should be
able to completely toss an entire AZ worth of instances, and their data,
without causing an outage. This is where you earn your sleep at night when
you're hosting on AWS.
2. Recovery from a botched deploy. Maybe a major version upgrade goes
sideways (we were bit hard during our beta when I messed up our 0.18 to
0.19 upgrade). A cheap snapshot lets you roll back your cluster state and
data with a couple filesystem operations.
3. Recovery from a total cluster loss by syncing back from S3. In practice,
we use our S3 backups more often to help a customer save time recovering
from an accidental deletion. And if we ever have an entire cluster outage,
asking all our customers to reindex is a big no-no for us.
When it's just your data, and you know how long it takes to reindex from
scratch, the cost-benefit analysis on reindexing versus restoring is up to
you. You probably want a restore from S3 to save you a couple hours in
order to be worth the effort.
Then again, this is all the kind of thing you should probably just do
anyway and assume your future self will appreciate. Or, you know, use
Bonsai 😉
--
Nick Zadrozny
Cofounder, One More Cloud
[websolr.com](http://websolr.com) [https://websolr.com/home](https://websolr.com/home) • [bonsai.io](http://bonsai.io) [http://bonsai.io/home](http://bonsai.io/home)
Hassle-free hosted full-text search,
powered by Apache Solr and Elasticsearch.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Bruno\_Miranda](https://discuss.elastic.co/u/Bruno_Miranda)
#### Post date: [February 21, 2013, 8:27pm UTC](https://discuss.elastic.co/t/index-backups-to-s3/10832/8 "2013-02-21T20:27:20Z")
Excellent answer. Thank you.
On Thursday, February 21, 2013 12:01:32 PM UTC-8, Nick Zadrozny wrote:
> On Wed, Feb 20, 2013 at 7:39 PM, Bruno Miranda \<[bru.m...@gmail.com](mailto:bru.m...@gmail.com)\
>
> > wrote:
>
> > I am wondering if backing up the index is necessary. If so, is S3 a good
> > place to put it?
>
> Here's what I would recommend. It's based on what we do for backups at
> [http://bonsai.io/](http://bonsai.io/) and is an alternative to EBS snapshots, which are a
> pretty reasonable approach if you're serving your data from a single EBS
> volume. (I think there are arguments for _not_ using EBS; another topic.)
>
> First, `cp -lr` your Elasticsearch data directory for a quick, cheap
> filesystem snapshot. This is one of those arcane bits of Unix knowledge
> that I learned once and understand intuitively, but probably would do a bad
> job explaining in detail, so consult man(1). Effectively, you get a cheap,
> instant copy and only pay the disk space for the delta as your original
> changes.
>
> Incidentally, `cp -lr` is cheap and useful enough that we use it to
> snapshot our data on every deploy, just in case.
>
> If your data _is_ on EBS, I would first `rsync -a` that snapshot over to
> the ephemeral store. This presumes you have enough space on your ephemeral
> store, which is a good constraint to consider when designing your cluster.
>
> An up-to-date ephemeral copy gives you some fairly cheap insurance when
> (not if) your EBS volume gets stuck. You can just change your data
> directory and restart the cluster. It should also save you some iops
> against your production EBS volume traffic while you're running your backup
> to S3.
>
> From the data snapshot, or rsync'd copy in your ephemeral store, you can
> use something akin to s3sync to send your data over to S3. We wrote a
> custom implementation; the backup gem that Karel linked looks reasonable
> too. We're also syncing into a "rolling window" of S3 buckets per daily
> backup, with a directory per host, since our main story for full backups is
> recovering from a customer's own accidental deletion.
>
> When considering availability in AWS, I have these priorities:
>
> 1. Replicate every index in at least two Availability Zones. You should be
> able to completely toss an entire AZ worth of instances, and their data,
> without causing an outage. This is where you earn your sleep at night when
> you're hosting on AWS.
>
> 2. Recovery from a botched deploy. Maybe a major version upgrade goes
> sideways (we were bit hard during our beta when I messed up our 0.18 to
> 0.19 upgrade). A cheap snapshot lets you roll back your cluster state and
> data with a couple filesystem operations.
>
> 3. Recovery from a total cluster loss by syncing back from S3. In
> practice, we use our S3 backups more often to help a customer save time
> recovering from an accidental deletion. And if we ever have an entire
> cluster outage, asking all our customers to reindex is a big no-no for us.
>
> When it's just your data, and you know how long it takes to reindex from
> scratch, the cost-benefit analysis on reindexing versus restoring is up to
> you. You probably want a restore from S3 to save you a couple hours in
> order to be worth the effort.
>
> Then again, this is all the kind of thing you should probably just do
> anyway and assume your future self will appreciate. Or, you know, use
> Bonsai 😉
>
> --
> Nick Zadrozny
>
> Cofounder, One More Cloud
>
> [websolr.com](http://websolr.com) [https://websolr.com/home](https://websolr.com/home) • [bonsai.io](http://bonsai.io) [http://bonsai.io/home](http://bonsai.io/home)
>
> Hassle-free hosted full-text search,
> powered by Apache Solr and Elasticsearch.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 2:50am UTC](https://discuss.elastic.co/t/index-backups-to-s3/10832/9 "2017-07-06T02:50:11Z")