# How to get a local ES to do an async copy over (replicate-over) of Prod ES indices? **URL:** \ **Category:** Elasticsearch\ **Created:** [March 15, 2012, 4:42pm UTC](https://discuss.elastic.co/t/how-to-get-a-local-es-to-do-an-async-copy-over-replicate-over-of-prod-es-indices/7018 "2012-03-15T16:42:23Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![pulkitsinghal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pulkitsinghal/32/802_2.png) [@pulkitsinghal](https://discuss.elastic.co/u/pulkitsinghal)\ **Post date:** [March 15, 2012, 4:42pm UTC](https://discuss.elastic.co/t/how-to-get-a-local-es-to-do-an-async-copy-over-replicate-over-of-prod-es-indices/7018/1 "2012-03-15T16:42:23Z")
I know that it is possible to start the index without any replicas, and add them at a later point: [http://www.elasticsearch.org/guide/reference/api/admin-indices-update-settings.html](http://www.elasticsearch.org/guide/reference/api/admin-indices-update-settings.html) But doing so in the simplest manner would only cause the replicated copy to exist on the cloud where my production ES instance lives. Can anyone tell me how to start a local ES instance and perform an async replication of the data from the cloud over to my machine? If replication is not the right term then lets call it as data dump or whatever else fits the bill here. Any thoughts? ---
**Author:** ![gearond](https://avatars.discourse-cdn.com/v4/letter/g/e68b1a/32.png) [@gearond](https://discuss.elastic.co/u/gearond)\ **Post date:** [March 15, 2012, 5:21pm UTC](https://discuss.elastic.co/t/how-to-get-a-local-es-to-do-an-async-copy-over-replicate-over-of-prod-es-indices/7018/2 "2012-03-15T17:21:18Z")
Want to know this also. Posting to watch replies. ---
**Author:** ![pulkitsinghal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pulkitsinghal/32/802_2.png) [@pulkitsinghal](https://discuss.elastic.co/u/pulkitsinghal)\ **Post date:** [March 15, 2012, 5:39pm UTC](https://discuss.elastic.co/t/how-to-get-a-local-es-to-do-an-async-copy-over-replicate-over-of-prod-es-indices/7018/3 "2012-03-15T17:39:40Z")
The following thread doesn't really answer my question (may be because I don't get how to set it up): [http://elasticsearch-users.115913.n3.nabble.com/how-to-dump-the-entire-contents-of-ES-td2758234.html](http://elasticsearch-users.115913.n3.nabble.com/how-to-dump-the-entire-contents-of-ES-td2758234.html) But Shay's comment, "copy of the data directory of each production cluster, and move it to development" makes sense. I'll try that out but in the mean time any other direct answers to my question would be most welcome. On Thu, Mar 15, 2012 at 12:21 PM, gearond [gearond@sbcglobal.net](mailto:gearond@sbcglobal.net) wrote: > Want to know this also. Posting to watch replies. > > -- > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/How-to-get-a-local-ES-to-do-an-async-copy-over-replicate-over-of-Prod-ES-indices-tp3829318p3829442.html](http://elasticsearch-users.115913.n3.nabble.com/How-to-get-a-local-ES-to-do-an-async-copy-over-replicate-over-of-Prod-ES-indices-tp3829318p3829442.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![pulkitsinghal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pulkitsinghal/32/802_2.png) [@pulkitsinghal](https://discuss.elastic.co/u/pulkitsinghal)\ **Post date:** [March 15, 2012, 5:57pm UTC](https://discuss.elastic.co/t/how-to-get-a-local-es-to-do-an-async-copy-over-replicate-over-of-prod-es-indices/7018/4 "2012-03-15T17:57:20Z")
Unfortunately the SCP command to get the data directory runs into problems early on: # ==== Sending file modes: C0664 51257065 \_5kr.fdt Sink: C0664 51257065 \_5kr.fdt \_5kr.fdt 46% 23MB 0.0KB/s - stalled - So I guess that's not a good route to go. On Thu, Mar 15, 2012 at 12:39 PM, Pulkit Singhal [pulkitsinghal@gmail.com](mailto:pulkitsinghal@gmail.com)wrote: > The following thread doesn't really answer my question (may be because I > don't get how to set it up): > > [http://elasticsearch-users.115913.n3.nabble.com/how-to-dump-the-entire-contents-of-ES-td2758234.html](http://elasticsearch-users.115913.n3.nabble.com/how-to-dump-the-entire-contents-of-ES-td2758234.html) > > But Shay's comment, "copy of the data directory of each production > cluster, and move it to development" makes sense. I'll try that out but in > the mean time any other direct answers to my question would be most welcome. > > On Thu, Mar 15, 2012 at 12:21 PM, gearond [gearond@sbcglobal.net](mailto:gearond@sbcglobal.net) wrote: > > > Want to know this also. Posting to watch replies. > > > > -- > > View this message in context: > > [http://elasticsearch-users.115913.n3.nabble.com/How-to-get-a-local-ES-to-do-an-async-copy-over-replicate-over-of-Prod-ES-indices-tp3829318p3829442.html](http://elasticsearch-users.115913.n3.nabble.com/How-to-get-a-local-ES-to-do-an-async-copy-over-replicate-over-of-Prod-ES-indices-tp3829318p3829442.html) > > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![pulkitsinghal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pulkitsinghal/32/802_2.png) [@pulkitsinghal](https://discuss.elastic.co/u/pulkitsinghal)\ **Post date:** [March 16, 2012, 12:58pm UTC](https://discuss.elastic.co/t/how-to-get-a-local-es-to-do-an-async-copy-over-replicate-over-of-prod-es-indices/7018/5 "2012-03-16T12:58:03Z")
So since its tougher to answer a question when something actually cannot happen, I take it that there is no way to start a local ES instance and perform an async replication of the data from the cloud over to my machine? On Thu, Mar 15, 2012 at 12:57 PM, Pulkit Singhal [pulkitsinghal@gmail.com](mailto:pulkitsinghal@gmail.com)wrote: > Unfortunately the SCP command to get the data directory runs into problems > early on: > > # ==== Sending file modes: C0664 51257065 \_5kr.fdt Sink: C0664 51257065 \_5kr.fdt \_5kr.fdt 46% 23MB 0.0KB/s - stalled - > > So I guess that's not a good route to go. > > On Thu, Mar 15, 2012 at 12:39 PM, Pulkit Singhal [pulkitsinghal@gmail.com](mailto:pulkitsinghal@gmail.com)wrote: > > > The following thread doesn't really answer my question (may be because I > > don't get how to set it up): > > > > [http://elasticsearch-users.115913.n3.nabble.com/how-to-dump-the-entire-contents-of-ES-td2758234.html](http://elasticsearch-users.115913.n3.nabble.com/how-to-dump-the-entire-contents-of-ES-td2758234.html) > > > > But Shay's comment, "copy of the data directory of each production > > cluster, and move it to development" makes sense. I'll try that out but in > > the mean time any other direct answers to my question would be most welcome. > > > > On Thu, Mar 15, 2012 at 12:21 PM, gearond [gearond@sbcglobal.net](mailto:gearond@sbcglobal.net) wrote: > > > > > Want to know this also. Posting to watch replies. > > > > > > -- > > > View this message in context: > > > [http://elasticsearch-users.115913.n3.nabble.com/How-to-get-a-local-ES-to-do-an-async-copy-over-replicate-over-of-Prod-ES-indices-tp3829318p3829442.html](http://elasticsearch-users.115913.n3.nabble.com/How-to-get-a-local-ES-to-do-an-async-copy-over-replicate-over-of-Prod-ES-indices-tp3829318p3829442.html) > > > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![Berkay\_Mollamustafao](https://avatars.discourse-cdn.com/v4/letter/b/22d042/32.png) [@Berkay\_Mollamustafao](https://discuss.elastic.co/u/Berkay_Mollamustafao)\ **Post date:** [March 16, 2012, 1:05pm UTC](https://discuss.elastic.co/t/how-to-get-a-local-es-to-do-an-async-copy-over-replicate-over-of-prod-es-indices/7018/6 "2012-03-16T13:05:27Z")
Right, there is no built-in mechanism to do that. You'd have to write code that reads from one and writes to the other yourself. Copying the files should have worked, not sure what that problem is. There has been some discussions about this in the list. If you search for "backup" in mailing list archive you should be able to find the discussions. Regards, Berkay Mollamustafaoglu mberkay on yahoo, google and skype On Fri, Mar 16, 2012 at 8:58 AM, Pulkit Singhal [pulkitsinghal@gmail.com](mailto:pulkitsinghal@gmail.com)wrote: > So since its tougher to answer a question when something actually cannot > happen, I take it that there is no way to start a local ES instance and > perform an async replication of the data from the cloud over to my machine? > > On Thu, Mar 15, 2012 at 12:57 PM, Pulkit Singhal [pulkitsinghal@gmail.com](mailto:pulkitsinghal@gmail.com)wrote: > > > Unfortunately the SCP command to get the data directory runs into > > problems early on: > > > > # ==== Sending file modes: C0664 51257065 \_5kr.fdt Sink: C0664 51257065 \_5kr.fdt \_5kr.fdt 46% 23MB 0.0KB/s - stalled - > > > > So I guess that's not a good route to go. > > > > On Thu, Mar 15, 2012 at 12:39 PM, Pulkit Singhal \<[pulkitsinghal@gmail.com](mailto:pulkitsinghal@gmail.com) > > > > > wrote: > > > > > The following thread doesn't really answer my question (may be because I > > > don't get how to set it up): > > > > > > [http://elasticsearch-users.115913.n3.nabble.com/how-to-dump-the-entire-contents-of-ES-td2758234.html](http://elasticsearch-users.115913.n3.nabble.com/how-to-dump-the-entire-contents-of-ES-td2758234.html) > > > > > > But Shay's comment, "copy of the data directory of each production > > > cluster, and move it to development" makes sense. I'll try that out but in > > > the mean time any other direct answers to my question would be most welcome. > > > > > > On Thu, Mar 15, 2012 at 12:21 PM, gearond [gearond@sbcglobal.net](mailto:gearond@sbcglobal.net) wrote: > > > > > > > Want to know this also. Posting to watch replies. > > > > > > > > -- > > > > View this message in context: > > > > [http://elasticsearch-users.115913.n3.nabble.com/How-to-get-a-local-ES-to-do-an-async-copy-over-replicate-over-of-Prod-ES-indices-tp3829318p3829442.html](http://elasticsearch-users.115913.n3.nabble.com/How-to-get-a-local-ES-to-do-an-async-copy-over-replicate-over-of-Prod-ES-indices-tp3829318p3829442.html) > > > > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [March 16, 2012, 2:07pm UTC](https://discuss.elastic.co/t/how-to-get-a-local-es-to-do-an-async-copy-over-replicate-over-of-prod-es-indices/7018/7 "2012-03-16T14:07:11Z")
On Fri, 2012-03-16 at 09:05 -0400, Berkay Mollamustafaoglu wrote: > Right, there is no built-in mechanism to do that. You'd have to write > code that reads from one and writes to the other yourself. And here's some code: [http://blogs.perl.org/users/clinton\_gormley/2011/04/elasticsearchpm-v036-now-with-extra-sugar.html](http://blogs.perl.org/users/clinton_gormley/2011/04/elasticsearchpm-v036-now-with-extra-sugar.html) > clint > Copying the files should have worked, not sure what that problem is. > There has been some discussions about this in the list. If you search > for "backup" in mailing list archive you should be able to find the > discussions. > > Regards, > Berkay Mollamustafaoglu > mberkay on yahoo, google and skype > > On Fri, Mar 16, 2012 at 8:58 AM, Pulkit Singhal > [pulkitsinghal@gmail.com](mailto:pulkitsinghal@gmail.com) wrote: > So since its tougher to answer a question when something > actually cannot happen, I take it that there is no way to > start a local ES instance and perform an async replication of > the data from the cloud over to my machine? > > ``` > On Thu, Mar 15, 2012 at 12:57 PM, Pulkit Singhal > wrote: > Unfortunately the SCP command to get the data > directory runs into problems early on: > > ==== > Sending file modes: C0664 51257065 _5kr.fdt > Sink: C0664 51257065 _5kr.fdt > _5kr.fdt 46% 23MB 0.0KB/s - stalled - > ==== > > So I guess that's not a good route to go. > > > On Thu, Mar 15, 2012 at 12:39 PM, Pulkit Singhal > wrote: > The following thread doesn't really answer my > question (may be because I don't get how to > set it up): > http://elasticsearch-users.115913.n3.nabble.com/how-to-dump-the-entire-contents-of-ES-td2758234.html > > But Shay's comment, "copy of the data > directory of each production cluster, and move > it to development" makes sense. I'll try that > out but in the mean time any other direct > answers to my question would be most welcome. > > > On Thu, Mar 15, 2012 at 12:21 PM, gearond > wrote: > Want to know this also. Posting to > watch replies. > > -- > View this message in context: > http://elasticsearch-users.115913.n3.nabble.com/How-to-get-a-local-ES-to-do-an-async-copy-over-replicate-over-of-Prod-ES-indices-tp3829318p3829442.html > Sent from the ElasticSearch Users > mailing list archive at Nabble.com. > > ``` ---
**Author:** ![pulkitsinghal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pulkitsinghal/32/802_2.png) [@pulkitsinghal](https://discuss.elastic.co/u/pulkitsinghal)\ **Post date:** [March 16, 2012, 3:48pm UTC](https://discuss.elastic.co/t/how-to-get-a-local-es-to-do-an-async-copy-over-replicate-over-of-prod-es-indices/7018/8 "2012-03-16T15:48:51Z")
Better way to make the transfer so that it can be picked up where its left off is to: a) zip up the data directory: tar -zcvf data.tar.gz /opt/elasticsearch/data b) use rync command that will let you pick up where you left off in case something messes up: rsync --rsh='ssh -i /users/xxx/.ec2/ec2.pem' --partial --progress ec2-user@XXX.XXX.XXX.XXX:/opt/elasticsearch/data.tar.gz ~/dev/elasticsearch/ On Thu, Mar 15, 2012 at 12:57 PM, Pulkit Singhal [pulkitsinghal@gmail.com](mailto:pulkitsinghal@gmail.com)wrote: > Unfortunately the SCP command to get the data directory runs into problems > early on: > > # ==== Sending file modes: C0664 51257065 \_5kr.fdt Sink: C0664 51257065 \_5kr.fdt \_5kr.fdt 46% 23MB 0.0KB/s - stalled - > > So I guess that's not a good route to go. > > On Thu, Mar 15, 2012 at 12:39 PM, Pulkit Singhal [pulkitsinghal@gmail.com](mailto:pulkitsinghal@gmail.com)wrote: > > > The following thread doesn't really answer my question (may be because I > > don't get how to set it up): > > > > [http://elasticsearch-users.115913.n3.nabble.com/how-to-dump-the-entire-contents-of-ES-td2758234.html](http://elasticsearch-users.115913.n3.nabble.com/how-to-dump-the-entire-contents-of-ES-td2758234.html) > > > > But Shay's comment, "copy of the data directory of each production > > cluster, and move it to development" makes sense. I'll try that out but in > > the mean time any other direct answers to my question would be most welcome. > > > > On Thu, Mar 15, 2012 at 12:21 PM, gearond [gearond@sbcglobal.net](mailto:gearond@sbcglobal.net) wrote: > > > > > Want to know this also. Posting to watch replies. > > > > > > -- > > > View this message in context: > > > [http://elasticsearch-users.115913.n3.nabble.com/How-to-get-a-local-ES-to-do-an-async-copy-over-replicate-over-of-Prod-ES-indices-tp3829318p3829442.html](http://elasticsearch-users.115913.n3.nabble.com/How-to-get-a-local-ES-to-do-an-async-copy-over-replicate-over-of-Prod-ES-indices-tp3829318p3829442.html) > > > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![pulkitsinghal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pulkitsinghal/32/802_2.png) [@pulkitsinghal](https://discuss.elastic.co/u/pulkitsinghal)\ **Post date:** [March 16, 2012, 3:52pm UTC](https://discuss.elastic.co/t/how-to-get-a-local-es-to-do-an-async-copy-over-replicate-over-of-prod-es-indices/7018/9 "2012-03-16T15:52:15Z")
That's truly awesome Clint! For me its unfortunate that I'm not using Perl (therefore my working knowledge gas atrophied) and this is also why I had trouble making the best of the "terms of endearment" slides. Not sure if I can learn enough to simply run with this but I'll try. Thansk for the great work. On Fri, Mar 16, 2012 at 9:07 AM, Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com)wrote: > On Fri, 2012-03-16 at 09:05 -0400, Berkay Mollamustafaoglu wrote: > > > Right, there is no built-in mechanism to do that. You'd have to write > > code that reads from one and writes to the other yourself. > > And here's some code: > > [ElasticSearch.pm v0.36, now with extra sugar | Clinton Gormley [blogs.perl.org]](http://blogs.perl.org/users/clinton_gormley/2011/04/elasticsearchpm-v036-now-with-extra-sugar.html) > > > > > clint > > > Copying the files should have worked, not sure what that problem is. > > There has been some discussions about this in the list. If you search > > for "backup" in mailing list archive you should be able to find the > > discussions. > > > > Regards, > > Berkay Mollamustafaoglu > > mberkay on yahoo, google and skype > > > > On Fri, Mar 16, 2012 at 8:58 AM, Pulkit Singhal > > [pulkitsinghal@gmail.com](mailto:pulkitsinghal@gmail.com) wrote: > > So since its tougher to answer a question when something > > actually cannot happen, I take it that there is no way to > > start a local ES instance and perform an async replication of > > the data from the cloud over to my machine? > > > > ``` > > On Thu, Mar 15, 2012 at 12:57 PM, Pulkit Singhal > > wrote: > > Unfortunately the SCP command to get the data > > directory runs into problems early on: > > > > ==== > > Sending file modes: C0664 51257065 _5kr.fdt > > Sink: C0664 51257065 _5kr.fdt > > _5kr.fdt 46% 23MB 0.0KB/s - stalled - > > ==== > > > > So I guess that's not a good route to go. > > > > On Thu, Mar 15, 2012 at 12:39 PM, Pulkit Singhal > > wrote: > > The following thread doesn't really answer my > > question (may be because I don't get how to > > set it up): > > > > ``` > > [http://elasticsearch-users.115913.n3.nabble.com/how-to-dump-the-entire-contents-of-ES-td2758234.html](http://elasticsearch-users.115913.n3.nabble.com/how-to-dump-the-entire-contents-of-ES-td2758234.html) > > > ``` > > But Shay's comment, "copy of the data > > directory of each production cluster, and move > > it to development" makes sense. I'll try that > > out but in the mean time any other direct > > answers to my question would be most welcome. > > > > On Thu, Mar 15, 2012 at 12:21 PM, gearond > > wrote: > > Want to know this also. Posting to > > watch replies. > > > > -- > > View this message in context: > > > > ``` > > [http://elasticsearch-users.115913.n3.nabble.com/How-to-get-a-local-ES-to-do-an-async-copy-over-replicate-over-of-Prod-ES-indices-tp3829318p3829442.html](http://elasticsearch-users.115913.n3.nabble.com/How-to-get-a-local-ES-to-do-an-async-copy-over-replicate-over-of-Prod-ES-indices-tp3829318p3829442.html) > > > ``` > > Sent from the ElasticSearch Users > > mailing list archive at Nabble.com. > > > > ``` ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [March 17, 2012, 10:24am UTC](https://discuss.elastic.co/t/how-to-get-a-local-es-to-do-an-async-copy-over-replicate-over-of-prod-es-indices/7018/10 "2012-03-17T10:24:25Z")
Yea, copying over the data location to your local environment should work, not sure why it failed. Usually, its recommended that you disable flush when doing so. Another option, which clinton provided an example for, is to use scroll search and reindex the data (or a portion of it, based on the query). On Fri, Mar 16, 2012 at 5:48 PM, Pulkit Singhal [pulkitsinghal@gmail.com](mailto:pulkitsinghal@gmail.com)wrote: > Better way to make the transfer so that it can be picked up where its left > off is to: > > a) zip up the data directory: > tar -zcvf data.tar.gz /opt/elasticsearch/data > > b) use rync command that will let you pick up where you left off in case > something messes up: > rsync --rsh='ssh -i /users/xxx/.ec2/ec2.pem' --partial --progress > ec2-user@XXX.XXX.XXX.XXX:/opt/elasticsearch/data.tar.gz > ~/dev/elasticsearch/ > > On Thu, Mar 15, 2012 at 12:57 PM, Pulkit Singhal [pulkitsinghal@gmail.com](mailto:pulkitsinghal@gmail.com)wrote: > > > Unfortunately the SCP command to get the data directory runs into > > problems early on: > > > > # ==== Sending file modes: C0664 51257065 \_5kr.fdt Sink: C0664 51257065 \_5kr.fdt \_5kr.fdt 46% 23MB 0.0KB/s - stalled - > > > > So I guess that's not a good route to go. > > > > On Thu, Mar 15, 2012 at 12:39 PM, Pulkit Singhal \<[pulkitsinghal@gmail.com](mailto:pulkitsinghal@gmail.com) > > > > > wrote: > > > > > The following thread doesn't really answer my question (may be because I > > > don't get how to set it up): > > > > > > [http://elasticsearch-users.115913.n3.nabble.com/how-to-dump-the-entire-contents-of-ES-td2758234.html](http://elasticsearch-users.115913.n3.nabble.com/how-to-dump-the-entire-contents-of-ES-td2758234.html) > > > > > > But Shay's comment, "copy of the data directory of each production > > > cluster, and move it to development" makes sense. I'll try that out but in > > > the mean time any other direct answers to my question would be most welcome. > > > > > > On Thu, Mar 15, 2012 at 12:21 PM, gearond [gearond@sbcglobal.net](mailto:gearond@sbcglobal.net) wrote: > > > > > > > Want to know this also. Posting to watch replies. > > > > > > > > -- > > > > View this message in context: > > > > [http://elasticsearch-users.115913.n3.nabble.com/How-to-get-a-local-ES-to-do-an-async-copy-over-replicate-over-of-Prod-ES-indices-tp3829318p3829442.html](http://elasticsearch-users.115913.n3.nabble.com/How-to-get-a-local-ES-to-do-an-async-copy-over-replicate-over-of-Prod-ES-indices-tp3829318p3829442.html) > > > > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:35am UTC](https://discuss.elastic.co/t/how-to-get-a-local-es-to-do-an-async-copy-over-replicate-over-of-prod-es-indices/7018/11 "2017-07-06T03:35:46Z")