# Gateway snapshots and thinking of a DR site
**URL:** https://discuss.elastic.co/t/gateway-snapshots-and-thinking-of-a-dr-site/4331
**Category:** Elasticsearch
**Created:** [May 3, 2011, 5:54am UTC](https://discuss.elastic.co/t/gateway-snapshots-and-thinking-of-a-dr-site/4331 "2011-05-03T05:54:59Z")
**Posts on this page:** 15
**Page:** 1
### Author:  [@Paul\_Smith](https://discuss.elastic.co/u/Paul_Smith)
#### Post date: [May 3, 2011, 5:54am UTC](https://discuss.elastic.co/t/gateway-snapshots-and-thinking-of-a-dr-site/4331/1 "2011-05-03T05:54:59Z")
Is there a way to have _2_ Gateways in ElasticSearch? Say, a local FS
Gateway for normal operations standard redundancy, plus, say, an S3 Gateway
for DR-style purposes.. ?
I'm guessing many people use ES to index database content, so wondering for
those people, who presumably (hopefully) have a DR strategy, how they're
ensuring their ES DR instance as close as possible ready to go to match the
DB that's in their DR too..?
I'd love to have someway of doing a nightly snapshot to, say, S3 or some
other remote location different from the standard gateway to give some
point-in-time recovery options.
any ideas?
Paul
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [May 3, 2011, 4:19pm UTC](https://discuss.elastic.co/t/gateway-snapshots-and-thinking-of-a-dr-site/4331/2 "2011-05-03T16:19:44Z")
There isn't a concept of 2 gateways, and (at least currently) there isn't a plan to do it because I would like to solve it in a different manner.
There are two aspects that are confused here. Backups (and recovering from them), in which case, backing up the data directory for each node works well for local gateway.
Another is multi site replication. This is solved today using messaging between the two sites and applying the same changes on two different ES clusters, and in the future, planned to be a built in option in ES.
On Tuesday, May 3, 2011 at 8:54 AM, Paul Smith wrote:
Is there a way to have _2_ Gateways in Elasticsearch? Say, a local FS Gateway for normal operations standard redundancy, plus, say, an S3 Gateway for DR-style purposes.. ?
> I'm guessing many people use ES to index database content, so wondering for those people, who presumably (hopefully) have a DR strategy, how they're ensuring their ES DR instance as close as possible ready to go to match the DB that's in their DR too..?
>
> I'd love to have someway of doing a nightly snapshot to, say, S3 or some other remote location different from the standard gateway to give some point-in-time recovery options.
>
> any ideas?
>
> Paul
---
### Author:  [@Paul\_Smith](https://discuss.elastic.co/u/Paul_Smith)
#### Post date: [May 3, 2011, 11:19pm UTC](https://discuss.elastic.co/t/gateway-snapshots-and-thinking-of-a-dr-site/4331/3 "2011-05-03T23:19:32Z")
On Wednesday, 4 May 2011, Shay Banon \<
> There are two aspects that are confused here. Backups (and recovering from them), in which case, backing up the data directory for each node works well for local gateway.
So with a local gateway there is no snapshotting available as I
understand from reading the docs. Is there a way to create a
quiescent period where one can guarantee the index on disk is flushed
and valid ready for copying? One needs to be able to create a valid
point in time snapshot of all shards before resuming on disk updates.
I think the same is required for the shared fs gateway option too.
This snapshots periodically for local data center redundancy but if
one needs to replicate this state to another DC location we need to be
able to pause any new snapshoting to get a quick valid directory tree
ready for replication.
This could be done either with a simple directory tree hard link
recurssion that Doug Cutting proposed way back near Lucene 1.4 if I
recall. Or if one has a SAN with Filesystem snapshotting then one can
use that.
Still needs a quiet period to ensure files are fsync'd and logically
consistent with respect to the index contents.
In the case of the local gateway would setting the flush frequency to
0 give this quiescent period for these sorts of operations? And then
resume flushing to continue the writes.
> Another is multi site replication. This is solved today using messaging between the two sites and applying the same changes on two different ES clusters, and in the future, planned to be a built in option in.
That will be awesome to have it builtin. However with any replication
strategy, planning for failure is a must. Imagine a replication stream
breaks somehow for whatever crazy reason, just total worst case say
the DR site blows up and one is forced to create it new. . One would
need to take a valid primary copy of the index to rebuild the DR and
get the replication stream going again.
In this scenario you need the same ability as above if you want to
keep the primary going still. Obviously shutting down ES is an option
here too but...
Has anyone had to replicate a large index to another cluster while the
primary is still up?
Paul Smith
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [May 4, 2011, 8:13pm UTC](https://discuss.elastic.co/t/gateway-snapshots-and-thinking-of-a-dr-site/4331/4 "2011-05-04T20:13:42Z")
Heya,
Regarding the copying option, yes, you can set teh translog settings (they can be changed dynamically now in 0.16) to high values, and then do the copying over (maybe flush before). This will make sure that no new Lucene commit point is written to disk.
As for cross (slow connection) DCs, the solution will have to handle complete failures, and the ability to actually copy over the whole indices back to the original data center. This will be handled automatically (in a similar fashion that a replica can recover from a primary shard when its allocated).
-shay.banon
On Wednesday, May 4, 2011 at 2:19 AM, Paul Smith wrote:
> On Wednesday, 4 May 2011, Shay Banon \<
>
> > There are two aspects that are confused here. Backups (and recovering from them), in which case, backing up the data directory for each node works well for local gateway.
>
> So with a local gateway there is no snapshotting available as I
> understand from reading the docs. Is there a way to create a
> quiescent period where one can guarantee the index on disk is flushed
> and valid ready for copying? One needs to be able to create a valid
> point in time snapshot of all shards before resuming on disk updates.
>
> I think the same is required for the shared fs gateway option too.
> This snapshots periodically for local data center redundancy but if
> one needs to replicate this state to another DC location we need to be
> able to pause any new snapshoting to get a quick valid directory tree
> ready for replication.
>
> This could be done either with a simple directory tree hard link
> recurssion that Doug Cutting proposed way back near Lucene 1.4 if I
> recall. Or if one has a SAN with Filesystem snapshotting then one can
> use that.
>
> Still needs a quiet period to ensure files are fsync'd and logically
> consistent with respect to the index contents.
>
> In the case of the local gateway would setting the flush frequency to
> 0 give this quiescent period for these sorts of operations? And then
> resume flushing to continue the writes.
>
> > Another is multi site replication. This is solved today using messaging between the two sites and applying the same changes on two different ES clusters, and in the future, planned to be a built in option in.
>
> That will be awesome to have it builtin. However with any replication
> strategy, planning for failure is a must. Imagine a replication stream
> breaks somehow for whatever crazy reason, just total worst case say
> the DR site blows up and one is forced to create it new. . One would
> need to take a valid primary copy of the index to rebuild the DR and
> get the replication stream going again.
>
> In this scenario you need the same ability as above if you want to
> keep the primary going still. Obviously shutting down ES is an option
> here too but...
>
> Has anyone had to replicate a large index to another cluster while the
> primary is still up?
>
> Paul Smith
---
### Author:  [@Paul\_Smith](https://discuss.elastic.co/u/Paul_Smith)
#### Post date: [May 4, 2011, 11:08pm UTC](https://discuss.elastic.co/t/gateway-snapshots-and-thinking-of-a-dr-site/4331/5 "2011-05-04T23:08:29Z")
On 5 May 2011 06:13, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
> Heya,
>
> Regarding the copying option, yes, you can set teh translog settings
> (they can be changed dynamically now in 0.16) to high values, and then do
> the copying over (maybe flush before). This will make sure that no new
> Lucene commit point is written to disk.
Excellent! So the 3 translog settings, if they are all changed to some
ridiculously large number to ensure none of the conditions triggers a flush,
and then a manual flush API call is done, it is safe to replicate the data
directory to another location, and then reset the settings.
> As for cross (slow connection) DCs, the solution will have to handle
> complete failures, and the ability to actually copy over the whole indices
> back to the original data center. This will be handled automatically (in a
> similar fashion that a replica can recover from a primary shard when its
> allocated).
That's one hell of a feature, can't wait. We have 1 DC in the Middle East,
with the DR currently in Sydney, and unfortunately the network link is
unreliable, high latency, generally a PITA. (we plan to address that with
another location, but it's a factor in our interim rollout).
Many thanks for taking the time to answer this, and so many other questions.
You're a machine to be able to keep this community supported AND all that
awesome development all at the same time. really appreciate it.
Paul
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [May 5, 2011, 10:31am UTC](https://discuss.elastic.co/t/gateway-snapshots-and-thinking-of-a-dr-site/4331/6 "2011-05-05T10:31:39Z")
On Thursday, May 5, 2011 at 2:08 AM, Paul Smith wrote:
> On 5 May 2011 06:13, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
>
> > Heya,
> >
> > Regarding the copying option, yes, you can set teh translog settings (they can be changed dynamically now in 0.16) to high values, and then do the copying over (maybe flush before). This will make sure that no new Lucene commit point is written to disk.
>
> Excellent! So the 3 translog settings, if they are all changed to some ridiculously large number to ensure none of the conditions triggers a flush, and then a manual flush API call is done, it is safe to replicate the data directory to another location, and then reset the settings.
> Yes. It can be simplified, we can add a disable flush flag that can be set dynamically. Open an issue?
>
> As for cross (slow connection) DCs, the solution will have to handle complete failures, and the ability to actually copy over the whole indices back to the original data center. This will be handled automatically (in a similar fashion that a replica can recover from a primary shard when its allocated).
>
> >
>
> That's one hell of a feature, can't wait. We have 1 DC in the Middle East, with the DR currently in Sydney, and unfortunately the network link is unreliable, high latency, generally a PITA. (we plan to address that with another location, but it's a factor in our interim rollout).
> Thats a big one :). I first going to start with support for single cluster across fast connected site (and having smart shard allocation that takes it into account).
>
> Many thanks for taking the time to answer this, and so many other questions. You're a machine to be able to keep this community supported AND all that awesome development all at the same time. really appreciate it.
> Thanks!
>
> Paul
---
### Author:  [@Paul\_Smith](https://discuss.elastic.co/u/Paul_Smith)
#### Post date: [May 5, 2011, 11:00am UTC](https://discuss.elastic.co/t/gateway-snapshots-and-thinking-of-a-dr-site/4331/7 "2011-05-05T11:00:56Z")
> Yes. It can be simplified, we can add a disable flush flag that can be set
> dynamically. Open an issue?
>
>
> Add \`index.translog.disable\_flush\` (defaults to \`false\`), which can be dynamical…ly updated using the \`update\_settings\` API. Note, this will only disable automatic flushing, explicit, api based flush will still flush.
>
> Based on a conversation on the user list:
>
> \> \> \> Regarding the copying option, yes, you can set teh translog settings (they can be changed dynamically now in 0.16) to high values, and then do the copying over (maybe flush before). This will make sure that no new Lucene commit point is written to disk.
> \> \>
> \> \> Excellent! So the 3 translog settings, if they are all changed to some ridiculously large number to ensure none of the conditions triggers a flush, and then a manual flush API call is done, it is safe to replicate the data directory to another location, and then reset the settings.
> \>
> \> Yes. It can be simplified, we can add a disable flush flag that can be set dynamically.
>
> The use case for a disabled flush is to allow a manual flush to occur, and \_then\_ some other process, like an rsync, or a fast recursive hard-link (and then an rsync perhaps) to facilitate secondary/tertiary/Disaster Recovery site replicant of the index and meta data.
thanks!
Paul
---
### Author:  [@Michel\_Conrad](https://discuss.elastic.co/u/Michel_Conrad)
#### Post date: [May 5, 2011, 4:36pm UTC](https://discuss.elastic.co/t/gateway-snapshots-and-thinking-of-a-dr-site/4331/8 "2011-05-05T16:36:44Z")
I would also love to see multi-DC replication.
So would it also be possible to have a cluster using the local storage
for speed, and the replica using ec2 storage for persistency?
Michel
On Thu, May 5, 2011 at 1:00 PM, Paul Smith [tallpsmith@gmail.com](mailto:tallpsmith@gmail.com) wrote:
> > Yes. It can be simplified, we can add a disable flush flag that can be set
> > dynamically. Open an issue?
>
> [Simplified Disable Flush operation · Issue #906 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/906)
> thanks!
> Paul
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [May 5, 2011, 9:36pm UTC](https://discuss.elastic.co/t/gateway-snapshots-and-thinking-of-a-dr-site/4331/9 "2011-05-05T21:36:54Z")
Which replica are you referring to? The replica "cluster"? What is ec2 storage, ebs/s3?
On Thursday, May 5, 2011 at 7:36 PM, Michel Conrad wrote:
> I would also love to see multi-DC replication.
> So would it also be possible to have a cluster using the local storage
> for speed, and the replica using ec2 storage for persistency?
>
> Michel
>
> On Thu, May 5, 2011 at 1:00 PM, Paul Smith [tallpsmith@gmail.com](mailto:tallpsmith@gmail.com) wrote:
>
> > > Yes. It can be simplified, we can add a disable flush flag that can be set
> > > dynamically. Open an issue?
> >
> > [Simplified Disable Flush operation · Issue #906 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/906)
> > thanks!
> > Paul
---
### Author:  [@Michel\_Conrad](https://discuss.elastic.co/u/Michel_Conrad)
#### Post date: [May 5, 2011, 10:03pm UTC](https://discuss.elastic.co/t/gateway-snapshots-and-thinking-of-a-dr-site/4331/10 "2011-05-05T22:03:20Z")
Oh I meant s3 instead of ec2. To rephrase my question would it be
possible to use an es cluster with local storage, which updates the
replications of the data on the s3 storage, so that in the event that
something goes wrong with the local storage one can recover
transparently from the s3 storage, creating a new local storage.
On Thu, May 5, 2011 at 11:36 PM, Shay Banon
[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
> Which replica are you referring to? The replica "cluster"? What is ec2
> storage, ebs/s3?
>
> On Thursday, May 5, 2011 at 7:36 PM, Michel Conrad wrote:
>
> I would also love to see multi-DC replication.
> So would it also be possible to have a cluster using the local storage
> for speed, and the replica using ec2 storage for persistency?
>
> Michel
>
> On Thu, May 5, 2011 at 1:00 PM, Paul Smith [tallpsmith@gmail.com](mailto:tallpsmith@gmail.com) wrote:
>
> Yes. It can be simplified, we can add a disable flush flag that can be set
> dynamically. Open an issue?
>
> [Simplified Disable Flush operation · Issue #906 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/906)
> thanks!
> Paul
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [May 5, 2011, 10:13pm UTC](https://discuss.elastic.co/t/gateway-snapshots-and-thinking-of-a-dr-site/4331/11 "2011-05-05T22:13:08Z")
The shared s3 gateway can be the main store data for the primary cluster. And if it fails, you can start another cluster which will use that data. The problem there is that it will take time to download all the data from s3 for the secondary cluster to be operational.
In general, a solution where a standby cluster makes sure it syncs against s3 is possible to implement, and then do the switch (which will be fast), but, the effort will be similar to having the standby cluster being synch'ed by the primary cluster (and there will be several sync modes, master - slave (in different modes), and master master).
Its all a bit up in the air. Currently, I want to first tackle a multi site with high speed connection with special shard allocation logic.
On Friday, May 6, 2011 at 1:03 AM, Michel Conrad wrote:
Oh I meant s3 instead of ec2. To rephrase my question would it be
> possible to use an es cluster with local storage, which updates the
> replications of the data on the s3 storage, so that in the event that
> something goes wrong with the local storage one can recover
> transparently from the s3 storage, creating a new local storage.
>
> On Thu, May 5, 2011 at 11:36 PM, Shay Banon
> [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
>
> > Which replica are you referring to? The replica "cluster"? What is ec2
> > storage, ebs/s3?
> >
> > On Thursday, May 5, 2011 at 7:36 PM, Michel Conrad wrote:
> >
> > I would also love to see multi-DC replication.
> > So would it also be possible to have a cluster using the local storage
> > for speed, and the replica using ec2 storage for persistency?
> >
> > Michel
> >
> > On Thu, May 5, 2011 at 1:00 PM, Paul Smith [tallpsmith@gmail.com](mailto:tallpsmith@gmail.com) wrote:
> >
> > Yes. It can be simplified, we can add a disable flush flag that can be set
> > dynamically. Open an issue?
> >
> > [Simplified Disable Flush operation · Issue #906 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/906)
> > thanks!
> > Paul
---
### Author:  [@Michel\_Conrad](https://discuss.elastic.co/u/Michel_Conrad)
#### Post date: [May 6, 2011, 8:47am UTC](https://discuss.elastic.co/t/gateway-snapshots-and-thinking-of-a-dr-site/4331/12 "2011-05-06T08:47:19Z")
Multi site with automatic shard allocation surely sounds interesting.
As for the s3 gateway, am I understanding it correctly that when using
it, the indices are updated on the gateway and mirrored locally for
fast access.
After a full cluster shutdown and restart, will the data still be
available locally or do I have to download the whole indices again?
What about shard relocation, will the data be streamed
from the primary shard over the local network or from the s3 gateway.
Thanks very much for your answers and patches, that come always really quick.
Michel
On Fri, May 6, 2011 at 12:13 AM, Shay Banon
[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
> The shared s3 gateway can be the main store data for the primary cluster.
> And if it fails, you can start another cluster which will use that data. The
> problem there is that it will take time to download all the data from s3 for
> the secondary cluster to be operational.
> In general, a solution where a standby cluster makes sure it syncs against
> s3 is possible to implement, and then do the switch (which will be fast),
> but, the effort will be similar to having the standby cluster being synch'ed
> by the primary cluster (and there will be several sync modes, master - slave
> (in different modes), and master master).
> Its all a bit up in the air. Currently, I want to first tackle a multi site
> with high speed connection with special shard allocation logic.
>
> On Friday, May 6, 2011 at 1:03 AM, Michel Conrad wrote:
>
> Oh I meant s3 instead of ec2. To rephrase my question would it be
> possible to use an es cluster with local storage, which updates the
> replications of the data on the s3 storage, so that in the event that
> something goes wrong with the local storage one can recover
> transparently from the s3 storage, creating a new local storage.
>
> On Thu, May 5, 2011 at 11:36 PM, Shay Banon
> [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
>
> Which replica are you referring to? The replica "cluster"? What is ec2
> storage, ebs/s3?
>
> On Thursday, May 5, 2011 at 7:36 PM, Michel Conrad wrote:
>
> I would also love to see multi-DC replication.
> So would it also be possible to have a cluster using the local storage
> for speed, and the replica using ec2 storage for persistency?
>
> Michel
>
> On Thu, May 5, 2011 at 1:00 PM, Paul Smith [tallpsmith@gmail.com](mailto:tallpsmith@gmail.com) wrote:
>
> Yes. It can be simplified, we can add a disable flush flag that can be set
> dynamically. Open an issue?
>
> [Simplified Disable Flush operation · Issue #906 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/906)
> thanks!
> Paul
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [May 6, 2011, 3:26pm UTC](https://discuss.elastic.co/t/gateway-snapshots-and-thinking-of-a-dr-site/4331/13 "2011-05-06T15:26:44Z")
After a full cluster shutdown and startup, the primary shards will be allocated to nodes that have as much data similar to the data on s3, and only missing data is recovered from s3 (thats why its important to configure things like gateway.recover\_after\_nodes). Replicas are recovered from primary shards, not from s3, and also reuse local data if possible.
On Friday, May 6, 2011 at 11:47 AM, Michel Conrad wrote:
Multi site with automatic shard allocation surely sounds interesting.
> As for the s3 gateway, am I understanding it correctly that when using
> it, the indices are updated on the gateway and mirrored locally for
> fast access.
> After a full cluster shutdown and restart, will the data still be
> available locally or do I have to download the whole indices again?
> What about shard relocation, will the data be streamed
> from the primary shard over the local network or from the s3 gateway.
>
> Thanks very much for your answers and patches, that come always really quick.
>
> Michel
>
> On Fri, May 6, 2011 at 12:13 AM, Shay Banon
> [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
>
> > The shared s3 gateway can be the main store data for the primary cluster.
> > And if it fails, you can start another cluster which will use that data. The
> > problem there is that it will take time to download all the data from s3 for
> > the secondary cluster to be operational.
> > In general, a solution where a standby cluster makes sure it syncs against
> > s3 is possible to implement, and then do the switch (which will be fast),
> > but, the effort will be similar to having the standby cluster being synch'ed
> > by the primary cluster (and there will be several sync modes, master - slave
> > (in different modes), and master master).
> > Its all a bit up in the air. Currently, I want to first tackle a multi site
> > with high speed connection with special shard allocation logic.
> >
> > On Friday, May 6, 2011 at 1:03 AM, Michel Conrad wrote:
> >
> > Oh I meant s3 instead of ec2. To rephrase my question would it be
> > possible to use an es cluster with local storage, which updates the
> > replications of the data on the s3 storage, so that in the event that
> > something goes wrong with the local storage one can recover
> > transparently from the s3 storage, creating a new local storage.
> >
> > On Thu, May 5, 2011 at 11:36 PM, Shay Banon
> > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
> >
> > Which replica are you referring to? The replica "cluster"? What is ec2
> > storage, ebs/s3?
> >
> > On Thursday, May 5, 2011 at 7:36 PM, Michel Conrad wrote:
> >
> > I would also love to see multi-DC replication.
> > So would it also be possible to have a cluster using the local storage
> > for speed, and the replica using ec2 storage for persistency?
> >
> > Michel
> >
> > On Thu, May 5, 2011 at 1:00 PM, Paul Smith [tallpsmith@gmail.com](mailto:tallpsmith@gmail.com) wrote:
> >
> > Yes. It can be simplified, we can add a disable flush flag that can be set
> > dynamically. Open an issue?
> >
> > [Simplified Disable Flush operation · Issue #906 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/906)
> > thanks!
> > Paul
---
### Author:  [@Michel\_Conrad](https://discuss.elastic.co/u/Michel_Conrad)
#### Post date: [May 6, 2011, 3:44pm UTC](https://discuss.elastic.co/t/gateway-snapshots-and-thinking-of-a-dr-site/4331/14 "2011-05-06T15:44:14Z")
Thanks for the reply, that now finally makes it clear to me how the s3
storage can be used, I wasn't entirely sure how the recovery would be
done.
On Fri, May 6, 2011 at 5:26 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
> After a full cluster shutdown and startup, the primary shards will be
> allocated to nodes that have as much data similar to the data on s3, and
> only missing data is recovered from s3 (thats why its important to configure
> things like gateway.recover\_after\_nodes). Replicas are recovered from
> primary shards, not from s3, and also reuse local data if possible.
>
> On Friday, May 6, 2011 at 11:47 AM, Michel Conrad wrote:
>
> Multi site with automatic shard allocation surely sounds interesting.
>
> As for the s3 gateway, am I understanding it correctly that when using
> it, the indices are updated on the gateway and mirrored locally for
> fast access.
> After a full cluster shutdown and restart, will the data still be
> available locally or do I have to download the whole indices again?
> What about shard relocation, will the data be streamed
> from the primary shard over the local network or from the s3 gateway.
>
> Thanks very much for your answers and patches, that come always really
> quick.
>
> Michel
>
> On Fri, May 6, 2011 at 12:13 AM, Shay Banon
> [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
>
> The shared s3 gateway can be the main store data for the primary cluster.
> And if it fails, you can start another cluster which will use that data. The
> problem there is that it will take time to download all the data from s3 for
> the secondary cluster to be operational.
> In general, a solution where a standby cluster makes sure it syncs against
> s3 is possible to implement, and then do the switch (which will be fast),
> but, the effort will be similar to having the standby cluster being synch'ed
> by the primary cluster (and there will be several sync modes, master - slave
> (in different modes), and master master).
> Its all a bit up in the air. Currently, I want to first tackle a multi site
> with high speed connection with special shard allocation logic.
>
> On Friday, May 6, 2011 at 1:03 AM, Michel Conrad wrote:
>
> Oh I meant s3 instead of ec2. To rephrase my question would it be
> possible to use an es cluster with local storage, which updates the
> replications of the data on the s3 storage, so that in the event that
> something goes wrong with the local storage one can recover
> transparently from the s3 storage, creating a new local storage.
>
> On Thu, May 5, 2011 at 11:36 PM, Shay Banon
> [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
>
> Which replica are you referring to? The replica "cluster"? What is ec2
> storage, ebs/s3?
>
> On Thursday, May 5, 2011 at 7:36 PM, Michel Conrad wrote:
>
> I would also love to see multi-DC replication.
> So would it also be possible to have a cluster using the local storage
> for speed, and the replica using ec2 storage for persistency?
>
> Michel
>
> On Thu, May 5, 2011 at 1:00 PM, Paul Smith [tallpsmith@gmail.com](mailto:tallpsmith@gmail.com) wrote:
>
> Yes. It can be simplified, we can add a disable flush flag that can be set
> dynamically. Open an issue?
>
> [Simplified Disable Flush operation · Issue #906 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/906)
> thanks!
> Paul
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 4:06am UTC](https://discuss.elastic.co/t/gateway-snapshots-and-thinking-of-a-dr-site/4331/15 "2017-07-06T04:06:48Z")