# JSON Backups **URL:** \ **Category:** Elasticsearch\ **Created:** [February 5, 2013, 8:37pm UTC](https://discuss.elastic.co/t/json-backups/10630 "2013-02-05T20:37:07Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![brandon\_2](https://avatars.discourse-cdn.com/v4/letter/b/439d5e/32.png) [@brandon\_2](https://discuss.elastic.co/u/brandon_2)\ **Post date:** [February 5, 2013, 8:37pm UTC](https://discuss.elastic.co/t/json-backups/10630/1 "2013-02-05T20:37:07Z")
Hello - I've seen various solutions to this, but am wondering what additional solutions are out there. For DR purposes, we're looking into JSON backups of the ES data on a regular basis. Using Tire, and the Scroll / Scan API full backups can be accomplished as well as incremental / differentials if you specify a time range. The problem is, full backups are exceptionally slow. What are your solutions for doing backups to a 'raw' format? Thanks -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Vinicius\_Carvalho](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinicius_carvalho/32/770_2.png) [@Vinicius\_Carvalho](https://discuss.elastic.co/u/Vinicius_Carvalho)\ **Post date:** [February 5, 2013, 10:08pm UTC](https://discuss.elastic.co/t/json-backups/10630/2 "2013-02-05T22:08:48Z")
I would be very interested on this as well. Today we rely on the fact that we rebuild the index again every morning, so we keep the last one around, and in case of error we just flip the alias to the old one. But, as we move to an incremental approach, the backup per se I'm not worried, because we rely on the fact that we have replication, and to loose data, something really bad has to happen on a very large number of nodes. What I'm worried is the fact that we (developers) could push a bug into prod that can hurt the index (say we mess things and change a lot of documents wrong). I really would like to have the ability to have some sort of delayed backup (24 hrs old) of my current index. We are still spinning our heads on the best way of doing it here. Regards On Tuesday, February 5, 2013 3:37:07 PM UTC-5, [bra...@infochimps.com](mailto:bra...@infochimps.com) wrote: > Hello - > > I've seen various solutions to this, but am wondering what additional > solutions are out there. For DR purposes, we're looking into JSON backups > of the ES data on a regular basis. Using Tire, and the Scroll / Scan API > full backups can be accomplished as well as incremental / differentials if > you specify a time range. > > The problem is, full backups are exceptionally slow. What are your > solutions for doing backups to a 'raw' format? > > Thanks -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [February 5, 2013, 11:03pm UTC](https://discuss.elastic.co/t/json-backups/10630/3 "2013-02-05T23:03:25Z")
Note, afaik, the Elasticsearch team is working on a full-fledged backup/restore solution, using internal structures of the cluster to ensure reliability. For my personal devel purposes, I'm using my knapsack plugin [GitHub - jprante/elasticsearch-knapsack: Knapsack plugin is an import/export tool for Elasticsearch](https://github.com/jprante/elasticsearch-knapsack) to pack JSON \_source fields into tar.gz archives. These archives can also be indexed again (I use it to test various index settings and mappings and for post-processing in other apps). But, while exporting documents, the index must be silent (inactive). "Full backups are exceptionally slow" - I'm puzzled, can you give some numbers you observe? With Java API bulk indexing, I can go up to 5-6 MB/second on an average Intel Sandy Bridge single desktop PC node (some thousand docs per second). Jörg Am 05.02.13 23:08, schrieb Vinicius Carvalho: > I would be very interested on this as well. Today we rely on the fact > that we rebuild the index again every morning, so we keep the last one > around, and in case of error we just flip the alias to the old one. > > But, as we move to an incremental approach, the backup per se I'm not > worried, because we rely on the fact that we have replication, and to > loose data, something really bad has to happen on a very large number > of nodes. > > What I'm worried is the fact that we (developers) could push a bug > into prod that can hurt the index (say we mess things and change a lot > of documents wrong). I really would like to have the ability to have > some sort of delayed backup (24 hrs old) of my current index. > > We are still spinning our heads on the best way of doing it here. > > Regards > > On Tuesday, February 5, 2013 3:37:07 PM UTC-5, [bra...@infochimps.com](mailto:bra...@infochimps.com) > wrote: > > ``` > Hello - > > I've seen various solutions to this, but am wondering what > additional solutions are out there. For DR purposes, we're > looking into JSON backups of the ES data on a regular basis. Using > Tire, and the Scroll / Scan API full backups can be accomplished > as well as incremental / differentials if you specify a time range. > > The problem is, full backups are exceptionally slow. What are > your solutions for doing backups to a 'raw' format? > > Thanks > > ``` > > -- > You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![brandon\_2](https://avatars.discourse-cdn.com/v4/letter/b/439d5e/32.png) [@brandon\_2](https://discuss.elastic.co/u/brandon_2)\ **Post date:** [February 5, 2013, 11:09pm UTC](https://discuss.elastic.co/t/json-backups/10630/4 "2013-02-05T23:09:43Z")
Hi Jörg - I've looked at knapsack as well, unfortunately the indexes are not silent. I'll look at it again to see if I can utilize it somehow. Extremely slow may have been a little too strong; dumping ~8 million documents to a GZipped JSON file took approximately 2.5 hours. This in itself is not a problem, but as the number of documents grow full backups will be come increasingly cumbersome. Thanks, On Tuesday, February 5, 2013 5:03:25 PM UTC-6, Jörg Prante wrote: > Note, afaik, the Elasticsearch team is working on a full-fledged > backup/restore solution, using internal structures of the cluster to > ensure reliability. > > For my personal devel purposes, I'm using my knapsack plugin > [GitHub - jprante/elasticsearch-knapsack: Knapsack plugin is an import/export tool for Elasticsearch](https://github.com/jprante/elasticsearch-knapsack) to pack JSON \_source > fields into tar.gz archives. These archives can also be indexed again (I > use it to test various index settings and mappings and for > post-processing in other apps). But, while exporting documents, the > index must be silent (inactive). > > "Full backups are exceptionally slow" - I'm puzzled, can you give some > numbers you observe? With Java API bulk indexing, I can go up to 5-6 > MB/second on an average Intel Sandy Bridge single desktop PC node (some > thousand docs per second). > > Jörg > > Am 05.02.13 23:08, schrieb Vinicius Carvalho: > > > I would be very interested on this as well. Today we rely on the fact > > that we rebuild the index again every morning, so we keep the last one > > around, and in case of error we just flip the alias to the old one. > > > > But, as we move to an incremental approach, the backup per se I'm not > > worried, because we rely on the fact that we have replication, and to > > loose data, something really bad has to happen on a very large number > > of nodes. > > > > What I'm worried is the fact that we (developers) could push a bug > > into prod that can hurt the index (say we mess things and change a lot > > of documents wrong). I really would like to have the ability to have > > some sort of delayed backup (24 hrs old) of my current index. > > > > We are still spinning our heads on the best way of doing it here. > > > > Regards > > > > On Tuesday, February 5, 2013 3:37:07 PM UTC-5, [bra...@infochimps.com](mailto:bra...@infochimps.com) > > wrote: > > > > ``` > > Hello - > > > > I've seen various solutions to this, but am wondering what > > additional solutions are out there. For DR purposes, we're > > looking into JSON backups of the ES data on a regular basis. Using > > Tire, and the Scroll / Scan API full backups can be accomplished > > as well as incremental / differentials if you specify a time range. > > > > The problem is, full backups are exceptionally slow. What are > > your solutions for doing backups to a 'raw' format? > > > > Thanks > > > > ``` > > > > -- > > You received this message because you are subscribed to the Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:52am UTC](https://discuss.elastic.co/t/json-backups/10630/5 "2017-07-06T02:52:52Z")