# So, with the deprecation of S3 gateway, what is the current best approach to cluster persistence? **URL:** \ **Category:** Elasticsearch\ **Created:** [January 20, 2013, 3:15pm UTC](https://discuss.elastic.co/t/so-with-the-deprecation-of-s3-gateway-what-is-the-current-best-approach-to-cluster-persistence/10428 "2013-01-20T15:15:08Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![haizaar](https://avatars.discourse-cdn.com/v4/letter/h/3d9bf3/32.png) [@haizaar](https://discuss.elastic.co/u/haizaar)\ **Post date:** [January 20, 2013, 3:15pm UTC](https://discuss.elastic.co/t/so-with-the-deprecation-of-s3-gateway-what-is-the-current-best-approach-to-cluster-persistence/10428/1 "2013-01-20T15:15:08Z")
Hello, Since S3 Gateway has been officially deprecated, how should one maintain cluster persistence while running on Amazon cloud? I can think of 1. Once in a while, flush; disable\_flush on a node. 2. rsync index data to backup folder 3. sync backup folder to S3 on background, to the folder named under the node name. 4. Do that for every node That way one can have the latest snapshot of the cluster backed up to S3 up to the latest snapshot point. But if I have 20 nodes in the cluster, restoring it from scratch will be a lot of manual work. Another way I can see: 1. Run on EBS 2. Periodically flush/disable\_flush on a node 3. sync 4. create EBS snapshot 5. enable\_flush But still 1. Need to take care of older snapshots pruning 2. Resting still looks like manual pane. So what is the advised practice of running multi-node cluster on AWS with ability to recover from cluster sudden death? Can I still go with S3 gateway if I'll take particular precautions that someone can outline? Best regards, Zaar -- ---
**Author:** ![karmi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karmi/32/44951_2.png) [@karmi](https://discuss.elastic.co/u/karmi)\ **Post date:** [January 21, 2013, 8:22am UTC](https://discuss.elastic.co/t/so-with-the-deprecation-of-s3-gateway-what-is-the-current-best-approach-to-cluster-persistence/10428/2 "2013-01-21T08:22:46Z")
> Since S3 Gateway has been officially deprecated, how should one maintain > cluster persistence while running on Amazon cloud? EBS :), preferably IOPS Another way I can see: > I believe you don't have to disable flush with EBS snapshots, since they should be point-in-time snapshots. Some documentation is at [1]. So what is the advised practice of running multi-node cluster on AWS with > ability to recover from cluster sudden death? Create EBS snapshots in intervals which make economical sense to you (hour,day,week?). For planning your disaster recovery plan, just hard-terminate your nodes. Then, create new EBS volumes from your snapshots, launch new EC2 instances, attach those volumes, voila cluster should be fine. It's a good idea to have a process like this automated, of course. See the Elasticsearch Chef tutorial [2] for a detailed walktrough of one possibility. Need to take care of older snapshots pruning > With a good library such as Fog for Ruby [3], it's really easy to have it all automated and nifty. There are many scripts on the internet for inspiration: [automate ebs snapshots fog - Google Search](https://www.google.com/search?q=automate+ebs+snapshots+fog&oq=automate+ebs+snapshots+fog) > Can I still go with S3 gateway if I'll take particular precautions that > someone can outline? The official Elasticsearch advice, and my own advice, based on personal experience is _don't do that_. There _are_ adventurous people who enjoy some thrill, though ๐Ÿ™‚ Karel [1] > [2] [Elasticsearch Platform โ€” Find real-time answers at scale | Elastic](http://www.elasticsearch.org/tutorials/2012/03/21/deploying-elasticsearch-with-chef-solo.html) [3] [fog - Getting Started](http://fog.io/about/getting_started.html) -- ---
**Author:** ![haizaar](https://avatars.discourse-cdn.com/v4/letter/h/3d9bf3/32.png) [@haizaar](https://discuss.elastic.co/u/haizaar)\ **Post date:** [January 21, 2013, 9:36am UTC](https://discuss.elastic.co/t/so-with-the-deprecation-of-s3-gateway-what-is-the-current-best-approach-to-cluster-persistence/10428/3 "2013-01-21T09:36:53Z")
Karel, thank you for your definite points. The path is clear now. Zaar On Monday, January 21, 2013 10:22:46 AM UTC+2, Karel Minaล™รญk wrote: > Since S3 Gateway has been officially deprecated, how should one maintain > > > cluster persistence while running on Amazon cloud? > > EBS :), preferably IOPS > > Another way I can see: > > > > > I believe you don't have to disable flush with EBS snapshots, since they > should be point-in-time snapshots. Some documentation is at [1]. > > So what is the advised practice of running multi-node cluster on AWS with > > > ability to recover from cluster sudden death? > > Create EBS snapshots in intervals which make economical sense to you > (hour,day,week?). For planning your disaster recovery plan, just > hard-terminate your nodes. Then, create new EBS volumes from your > snapshots, launch new EC2 instances, attach those volumes, voila cluster > should be fine. > > It's a good idea to have a process like this automated, of course. See the > Elasticsearch Chef tutorial [2] for a detailed walktrough of one > possibility. > > Need to take care of older snapshots pruning > > > > > With a good library such as Fog for Ruby [3], it's really easy to have it > all automated and nifty. There are many scripts on the internet for > inspiration: > [automate ebs snapshots fog - Google Search](https://www.google.com/search?q=automate+ebs+snapshots+fog&oq=automate+ebs+snapshots+fog) > > > Can I still go with S3 gateway if I'll take particular precautions that > > someone can outline? > > The official Elasticsearch advice, and my own advice, based on personal > experience is _don't do that_. There _are_ adventurous people who enjoy > some thrill, though ๐Ÿ™‚ > > Karel > > [1] > [Amazon EBS, snapshots as incremental backups - Stack Overflow](http://stackoverflow.com/questions/6469556/amazon-ebs-snapshots-as-incremental-backups) > [2] > [Elasticsearch Platform โ€” Find real-time answers at scale | Elastic](http://www.elasticsearch.org/tutorials/2012/03/21/deploying-elasticsearch-with-chef-solo.html) > [3] [fog - Getting Started](http://fog.io/about/getting_started.html) -- ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [January 22, 2013, 5:46pm UTC](https://discuss.elastic.co/t/so-with-the-deprecation-of-s3-gateway-what-is-the-current-best-approach-to-cluster-persistence/10428/4 "2013-01-22T17:46:24Z")
Hi Karel, Have you tried the "create EBS snapshot whenever, without any ES flushing or pausing, and then recover ES from those EBS snapshots" approach yourself or anyone you know and reeeeally trust? ๐Ÿ™‚ ## Thanks, Otis Solr & Elasticsearch Support > **[Sematext | IT System Monitoring Tools for DevOps](https://sematext.com/)** > > IT system monitoring and management tools for DevOps who need 24x7 live visibility into their infrastructure. On Mon, Jan 21, 2013 at 3:22 AM, Karel Minaล™รญk [karel.minarik@gmail.com](mailto:karel.minarik@gmail.com)wrote: > Since S3 Gateway has been officially deprecated, how should one maintain > > > cluster persistence while running on Amazon cloud? > > EBS :), preferably IOPS > > Another way I can see: > > > > > I believe you don't have to disable flush with EBS snapshots, since they > should be point-in-time snapshots. Some documentation is at [1]. > > So what is the advised practice of running multi-node cluster on AWS with > > > ability to recover from cluster sudden death? > > Create EBS snapshots in intervals which make economical sense to you > (hour,day,week?). For planning your disaster recovery plan, just > hard-terminate your nodes. Then, create new EBS volumes from your > snapshots, launch new EC2 instances, attach those volumes, voila cluster > should be fine. > > It's a good idea to have a process like this automated, of course. See the > Elasticsearch Chef tutorial [2] for a detailed walktrough of one > possibility. > > Need to take care of older snapshots pruning > > > > > With a good library such as Fog for Ruby [3], it's really easy to have it > all automated and nifty. There are many scripts on the internet for > inspiration: > [Google Search](https://www.google.com/search?q=automate+ebs+snapshots+fog&oq=automate+ebs+snapshots+fog) > > > Can I still go with S3 gateway if I'll take particular precautions that > > someone can outline? > > The official Elasticsearch advice, and my own advice, based on personal > experience is _don't do that_. There _are_ adventurous people who enjoy > some thrill, though ๐Ÿ™‚ > > Karel > > [1] > [http://stackoverflow.com/questions/6469556/amazon-ebs-snapshots-as-incremental-backups](http://stackoverflow.com/questions/6469556/amazon-ebs-snapshots-as-incremental-backups) > [2] > [Elastic โ€” The Search AI Company | Elastic](http://www.elasticsearch.org/tutorials/2012/03/21/deploying-elasticsearch-with-chef-solo.html) > [3] [http://fog.io/about/getting\_started.html](http://fog.io/about/getting_started.html) > > -- -- ---
**Author:** ![karmi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karmi/32/44951_2.png) [@karmi](https://discuss.elastic.co/u/karmi)\ **Post date:** [January 22, 2013, 6:45pm UTC](https://discuss.elastic.co/t/so-with-the-deprecation-of-s3-gateway-what-is-the-current-best-approach-to-cluster-persistence/10428/5 "2013-01-22T18:45:27Z")
> > I believe you don't have to disable flush with EBS snapshots, since they should be point-in-time snapshots. Some documentation is at [1]. > > Have you tried the "create EBS snapshot whenever, without any ES flushing or pausing, and then recover ES from those EBS snapshots" approach yourself or anyone you know and reeeeally trust? ๐Ÿ™‚ As said, I "believe" EBS snapshots are point-in-time. I have, in fact, succesfully recovered from an EBS snapshot without previously disabling flush on a running system. But since it is hard to simulate all the variables and possibilities in play, disabling flush seems like a sane precausion to me. Karel -- ---
**Author:** ![haizaar](https://avatars.discourse-cdn.com/v4/letter/h/3d9bf3/32.png) [@haizaar](https://discuss.elastic.co/u/haizaar)\ **Post date:** [January 22, 2013, 8:41pm UTC](https://discuss.elastic.co/t/so-with-the-deprecation-of-s3-gateway-what-is-the-current-best-approach-to-cluster-persistence/10428/6 "2013-01-22T20:41:32Z")
Karel, Do I have to also issue "flush" after "disable\_flush" to make sure that everything is in sync? Zaar On 22 ื‘ื™ื ื• 2013, at 20:45, "Karel Minaล™รญk" [karel.minarik@gmail.com](mailto:karel.minarik@gmail.com) wrote: > > > I believe you don't have to disable flush with EBS snapshots, since they should be point-in-time snapshots. Some documentation is at [1]. > > > Have you tried the "create EBS snapshot whenever, without any ES flushing or pausing, and then recover ES from those EBS snapshots" approach yourself or anyone you know and reeeeally trust? ๐Ÿ™‚ > > As said, I "believe" EBS snapshots are point-in-time. I have, in fact, succesfully recovered from an EBS snapshot without previously disabling flush on a running system. But since it is hard to simulate all the variables and possibilities in play, disabling flush seems like a sane precausion to me. > > Karel > > -- -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:55am UTC](https://discuss.elastic.co/t/so-with-the-deprecation-of-s3-gateway-what-is-the-current-best-approach-to-cluster-persistence/10428/7 "2017-07-06T02:55:08Z")