# Is it possible to copy a gateway snapshot between clusters? **URL:** \ **Category:** Elasticsearch\ **Created:** [August 4, 2010, 7:46pm UTC](https://discuss.elastic.co/t/is-it-possible-to-copy-a-gateway-snapshot-between-clusters/3189 "2010-08-04T19:46:54Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Grant\_Rodgers](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/grant_rodgers/32/3204_2.png) [@Grant\_Rodgers](https://discuss.elastic.co/u/Grant_Rodgers)\ **Post date:** [August 4, 2010, 7:46pm UTC](https://discuss.elastic.co/t/is-it-possible-to-copy-a-gateway-snapshot-between-clusters/3189/1 "2010-08-04T19:46:54Z")
We would like to copy the gateway snapshot from a production cluster to a development cluster. Since our production gateway is on s3, copying indices to development could take several minutes, during which the production cluster would still be snapshotting. Is this safe? If not, could we disable snapshotting temporarily while the copy is happening? Also, is it possible to copy an s3 gateway to a filesystem gateway? We are running ES 0.9. Thanks, Grant ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 4, 2010, 7:50pm UTC](https://discuss.elastic.co/t/is-it-possible-to-copy-a-gateway-snapshot-between-clusters/3189/2 "2010-08-04T19:50:42Z")
Yes, you can safely copy over the gateway data, either to another s3 or filesystem, even while its snapshotting. -shay.banon On Wed, Aug 4, 2010 at 10:46 PM, Grant Rodgers [grantr@gmail.com](mailto:grantr@gmail.com) wrote: > We would like to copy the gateway snapshot from a production cluster > to a development cluster. Since our production gateway is on s3, > copying indices to development could take several minutes, during > which the production cluster would still be snapshotting. Is this > safe? If not, could we disable snapshotting temporarily while the copy > is happening? > > Also, is it possible to copy an s3 gateway to a filesystem gateway? > > We are running ES 0.9. > > Thanks, > Grant ---
**Author:** ![Grant\_Rodgers](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/grant_rodgers/32/3204_2.png) [@Grant\_Rodgers](https://discuss.elastic.co/u/Grant_Rodgers)\ **Post date:** [August 4, 2010, 7:56pm UTC](https://discuss.elastic.co/t/is-it-possible-to-copy-a-gateway-snapshot-between-clusters/3189/3 "2010-08-04T19:56:18Z")
Great! Is it also possible to copy only certain indices? We would like to be able to copy only the metadata directory and index directories for the indices we care about. I guess the question is, can the gateway recovery handle missing indices in the gateway snapshot? Thanks, Grant On Aug 4, 12:50 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > Yes, you can safely copy over the gateway data, either to another s3 or > filesystem, even while its snapshotting. > > -shay.banon > > On Wed, Aug 4, 2010 at 10:46 PM, Grant Rodgers [gra...@gmail.com](mailto:gra...@gmail.com) wrote: > > > We would like to copy the gateway snapshot from a production cluster > > to a development cluster. Since our production gateway is on s3, > > copying indices to development could take several minutes, during > > which the production cluster would still be snapshotting. Is this > > safe? If not, could we disable snapshotting temporarily while the copy > > is happening? > > > Also, is it possible to copy an s3 gateway to a filesystem gateway? > > > We are running ES 0.9. > > > Thanks, > > Grant ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 4, 2010, 7:59pm UTC](https://discuss.elastic.co/t/is-it-possible-to-copy-a-gateway-snapshot-between-clusters/3189/4 "2010-08-04T19:59:12Z")
A simpler solution is to take the metadata-XXX file (its a json file) and remove to indices you don't want to use. It can also be hacked for example, to change the number of replicas and start the cluster again (though there will be an API for that). -shay.banon On Wed, Aug 4, 2010 at 10:56 PM, Grant Rodgers [grantr@gmail.com](mailto:grantr@gmail.com) wrote: > Great! Is it also possible to copy only certain indices? We would like > to be able to copy only the metadata directory and index directories > for the indices we care about. > > I guess the question is, can the gateway recovery handle missing > indices in the gateway snapshot? > > Thanks, > Grant > > On Aug 4, 12:50 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > Yes, you can safely copy over the gateway data, either to another s3 or > > filesystem, even while its snapshotting. > > > > -shay.banon > > > > On Wed, Aug 4, 2010 at 10:46 PM, Grant Rodgers [gra...@gmail.com](mailto:gra...@gmail.com) wrote: > > > > > We would like to copy the gateway snapshot from a production cluster > > > to a development cluster. Since our production gateway is on s3, > > > copying indices to development could take several minutes, during > > > which the production cluster would still be snapshotting. Is this > > > safe? If not, could we disable snapshotting temporarily while the copy > > > is happening? > > > > > Also, is it possible to copy an s3 gateway to a filesystem gateway? > > > > > We are running ES 0.9. > > > > > Thanks, > > > Grant ---
**Author:** ![Berkay\_Mollamustafao](https://avatars.discourse-cdn.com/v4/letter/b/22d042/32.png) [@Berkay\_Mollamustafao](https://discuss.elastic.co/u/Berkay_Mollamustafao)\ **Post date:** [August 4, 2010, 8:00pm UTC](https://discuss.elastic.co/t/is-it-possible-to-copy-a-gateway-snapshot-between-clusters/3189/5 "2010-08-04T20:00:54Z")
Nice !!! Regards, Berkay Mollamustafaoglu mberkay on yahoo, google and skype On Wed, Aug 4, 2010 at 3:59 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > A simpler solution is to take the metadata-XXX file (its a json file) and > remove to indices you don't want to use. It can also be hacked for example, > to change the number of replicas and start the cluster again (though there > will be an API for that). > > -shay.banon > > On Wed, Aug 4, 2010 at 10:56 PM, Grant Rodgers [grantr@gmail.com](mailto:grantr@gmail.com) wrote: > > > Great! Is it also possible to copy only certain indices? We would like > > to be able to copy only the metadata directory and index directories > > for the indices we care about. > > > > I guess the question is, can the gateway recovery handle missing > > indices in the gateway snapshot? > > > > Thanks, > > Grant > > > > On Aug 4, 12:50 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > > > Yes, you can safely copy over the gateway data, either to another s3 or > > > filesystem, even while its snapshotting. > > > > > > -shay.banon > > > > > > On Wed, Aug 4, 2010 at 10:46 PM, Grant Rodgers [gra...@gmail.com](mailto:gra...@gmail.com) > > > wrote: > > > > > > > We would like to copy the gateway snapshot from a production cluster > > > > to a development cluster. Since our production gateway is on s3, > > > > copying indices to development could take several minutes, during > > > > which the production cluster would still be snapshotting. Is this > > > > safe? If not, could we disable snapshotting temporarily while the copy > > > > is happening? > > > > > > > Also, is it possible to copy an s3 gateway to a filesystem gateway? > > > > > > > We are running ES 0.9. > > > > > > > Thanks, > > > > Grant ---
**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\ **Post date:** [August 4, 2010, 8:05pm UTC](https://discuss.elastic.co/t/is-it-possible-to-copy-a-gateway-snapshot-between-clusters/3189/6 "2010-08-04T20:05:30Z")
Shay, do you think you can elaborate more on this? I am surprised this is possible. My naive understanding is that if the snapshotting is going on then some files in gateway are changed, how it is then possible that the copy is consistent? Regards, Lukas On Wed, Aug 4, 2010 at 9:50 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > Yes, you can safely copy over the gateway data, either to another s3 or > filesystem, even while its snapshotting. > > -shay.banon > > On Wed, Aug 4, 2010 at 10:46 PM, Grant Rodgers [grantr@gmail.com](mailto:grantr@gmail.com) wrote: > > > We would like to copy the gateway snapshot from a production cluster > > to a development cluster. Since our production gateway is on s3, > > copying indices to development could take several minutes, during > > which the production cluster would still be snapshotting. Is this > > safe? If not, could we disable snapshotting temporarily while the copy > > is happening? > > > > Also, is it possible to copy an s3 gateway to a filesystem gateway? > > > > We are running ES 0.9. > > > > Thanks, > > Grant ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 4, 2010, 9:12pm UTC](https://discuss.elastic.co/t/is-it-possible-to-copy-a-gateway-snapshot-between-clusters/3189/7 "2010-08-04T21:12:08Z")
It basically boils down to how lucene works when storing the index, and the additional md5 checksum files elasticsearch produces for them. Basically, an index "version" is written to the gateway while another one exists, and only when its done being written to the gateway, then the "old" one is removed. The transaction log is an append only log, and keyed by the index version, so you just copy it over, and how many operations managed to get into it, you will get when you recovery. -shay.banon On Wed, Aug 4, 2010 at 11:05 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com) wrote: > Shay, > > do you think you can elaborate more on this? I am surprised this is > possible. My naive understanding is that if the snapshotting is going on > then some files in gateway are changed, how it is then possible that the > copy is consistent? > > Regards, > Lukas > > On Wed, Aug 4, 2010 at 9:50 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > > > Yes, you can safely copy over the gateway data, either to another s3 or > > filesystem, even while its snapshotting. > > > > -shay.banon > > > > On Wed, Aug 4, 2010 at 10:46 PM, Grant Rodgers [grantr@gmail.com](mailto:grantr@gmail.com) wrote: > > > > > We would like to copy the gateway snapshot from a production cluster > > > to a development cluster. Since our production gateway is on s3, > > > copying indices to development could take several minutes, during > > > which the production cluster would still be snapshotting. Is this > > > safe? If not, could we disable snapshotting temporarily while the copy > > > is happening? > > > > > > Also, is it possible to copy an s3 gateway to a filesystem gateway? > > > > > > We are running ES 0.9. > > > > > > Thanks, > > > Grant ---
**Author:** ![Grant\_Rodgers](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/grant_rodgers/32/3204_2.png) [@Grant\_Rodgers](https://discuss.elastic.co/u/Grant_Rodgers)\ **Post date:** [August 5, 2010, 12:03am UTC](https://discuss.elastic.co/t/is-it-possible-to-copy-a-gateway-snapshot-between-clusters/3189/8 "2010-08-05T00:03:36Z")
This is yet another example of the consistently elegant and impressive architecture of elasticsearch. Keep up the good work Shay. I for one am amazed by the rapid pace of development! On Aug 4, 2:12 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > It basically boils down to how lucene works when storing the index, and the > additional md5 checksum files elasticsearch produces for them. Basically, an > index "version" is written to the gateway while another one exists, and only > when its done being written to the gateway, then the "old" one is removed. > The transaction log is an append only log, and keyed by the index version, > so you just copy it over, and how many operations managed to get into it, > you will get when you recovery. > > -shay.banon > > On Wed, Aug 4, 2010 at 11:05 PM, Lukáš Vlček [lukas.vl...@gmail.com](mailto:lukas.vl...@gmail.com) wrote: > > > Shay, > > > do you think you can elaborate more on this? I am surprised this is > > possible. My naive understanding is that if the snapshotting is going on > > then some files in gateway are changed, how it is then possible that the > > copy is consistent? > > > Regards, > > Lukas > > > On Wed, Aug 4, 2010 at 9:50 PM, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com)wrote: > > > > Yes, you can safely copy over the gateway data, either to another s3 or > > > filesystem, even while its snapshotting. > > > > -shay.banon > > > > On Wed, Aug 4, 2010 at 10:46 PM, Grant Rodgers [gra...@gmail.com](mailto:gra...@gmail.com) wrote: > > > > > We would like to copy the gateway snapshot from a production cluster > > > > to a development cluster. Since our production gateway is on s3, > > > > copying indices to development could take several minutes, during > > > > which the production cluster would still be snapshotting. Is this > > > > safe? If not, could we disable snapshotting temporarily while the copy > > > > is happening? > > > > > Also, is it possible to copy an s3 gateway to a filesystem gateway? > > > > > We are running ES 0.9. > > > > > Thanks, > > > > Grant ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 5, 2010, 7:15am UTC](https://discuss.elastic.co/t/is-it-possible-to-copy-a-gateway-snapshot-between-clusters/3189/9 "2010-08-05T07:15:39Z")
thanks! On Thu, Aug 5, 2010 at 3:03 AM, Grant Rodgers [grantr@gmail.com](mailto:grantr@gmail.com) wrote: > This is yet another example of the consistently elegant and impressive > architecture of elasticsearch. Keep up the good work Shay. I for one > am amazed by the rapid pace of development! > > On Aug 4, 2:12 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > It basically boils down to how lucene works when storing the index, and > > the > > additional md5 checksum files elasticsearch produces for them. Basically, > > an > > index "version" is written to the gateway while another one exists, and > > only > > when its done being written to the gateway, then the "old" one is > > removed. > > The transaction log is an append only log, and keyed by the index > > version, > > so you just copy it over, and how many operations managed to get into it, > > you will get when you recovery. > > > > -shay.banon > > > > On Wed, Aug 4, 2010 at 11:05 PM, Lukáš Vlček [lukas.vl...@gmail.com](mailto:lukas.vl...@gmail.com) > > wrote: > > > > > Shay, > > > > > do you think you can elaborate more on this? I am surprised this is > > > possible. My naive understanding is that if the snapshotting is going > > > on > > > then some files in gateway are changed, how it is then possible that > > > the > > > copy is consistent? > > > > > Regards, > > > Lukas > > > > > On Wed, Aug 4, 2010 at 9:50 PM, Shay Banon \< > > > [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com)\>wrote: > > > > > > Yes, you can safely copy over the gateway data, either to another s3 > > > > or > > > > filesystem, even while its snapshotting. > > > > > > -shay.banon > > > > > > On Wed, Aug 4, 2010 at 10:46 PM, Grant Rodgers [gra...@gmail.com](mailto:gra...@gmail.com) > > > > wrote: > > > > > > > We would like to copy the gateway snapshot from a production cluster > > > > > to a development cluster. Since our production gateway is on s3, > > > > > copying indices to development could take several minutes, during > > > > > which the production cluster would still be snapshotting. Is this > > > > > safe? If not, could we disable snapshotting temporarily while the > > > > > copy > > > > > is happening? > > > > > > > Also, is it possible to copy an s3 gateway to a filesystem gateway? > > > > > > > We are running ES 0.9. > > > > > > > Thanks, > > > > > Grant ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:21am UTC](https://discuss.elastic.co/t/is-it-possible-to-copy-a-gateway-snapshot-between-clusters/3189/10 "2017-07-06T04:21:07Z")