# Data(Cluster) migration in seperate networks **URL:** \ **Category:** Elasticsearch\ **Created:** [September 18, 2013, 2:55pm UTC](https://discuss.elastic.co/t/data-cluster-migration-in-seperate-networks/13672 "2013-09-18T14:55:09Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Derry\_O\_Sullivan](https://avatars.discourse-cdn.com/v4/letter/d/7bcc69/32.png) [@Derry\_O\_Sullivan](https://discuss.elastic.co/u/Derry_O_Sullivan)\ **Post date:** [September 18, 2013, 2:55pm UTC](https://discuss.elastic.co/t/data-cluster-migration-in-seperate-networks/13672/1 "2013-09-18T14:55:09Z")
Hi all, I'm working on some scalability planning at the moment for our ES deployment and looking at a hypothetical situation where we want to move our es data from one cluster (e.g. private network or cloud (ec2/rackspace/other)) to another. If i wanted to migrate from this private cluster (e.g. in my own office subnet) to e.g. EC2 cluster, is it possible to do 'auto' migration (e.g. set up unicast discovery on the ec2 cluster to 'find' the older instance on my private network and allow the replica's to be created on EC2). e.g. let ES manage the migration. I understand that i may need to increase the number of replica's and make sure i have enough nodes in the new cluster to meet that requirement I understand that this option will have latency issues (i.e. would need to keep the servers physically close/etc) but would there be any other issues? Once the data has been replicated successfully (i.e. enough replica's exist on new cluster), the old server cluster could then be powered down with the data having been migrated? Thanks, Derry -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [September 18, 2013, 9:50pm UTC](https://discuss.elastic.co/t/data-cluster-migration-in-seperate-networks/13672/2 "2013-09-18T21:50:33Z")
Hey, have you seen the scan search type at > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. is there anything speaking against that in your setup? Just wondering why you want to have the complexity of connecting clusters across the internet (with all the stability issues the internet offers). --Alex On Wed, Sep 18, 2013 at 4:55 PM, Derry O' Sullivan [derryos@gmail.com](mailto:derryos@gmail.com)wrote: > Hi all, > > I'm working on some scalability planning at the moment for our ES > deployment and looking at a hypothetical situation where we want to move > our es data from one cluster (e.g. private network or cloud > (ec2/rackspace/other)) to another. > > If i wanted to migrate from this private cluster (e.g. in my own office > subnet) to e.g. EC2 cluster, is it possible to do 'auto' migration (e.g. > set up unicast discovery on the ec2 cluster to 'find' the older instance on > my private network and allow the replica's to be created on EC2). e.g. let > ES manage the migration. I understand that i may need to increase the > number of replica's and make sure i have enough nodes in the new cluster to > meet that requirement > > I understand that this option will have latency issues (i.e. would need to > keep the servers physically close/etc) but would there be any other issues? > Once the data has been replicated successfully (i.e. enough replica's exist > on new cluster), the old server cluster could then be powered down with the > data having been migrated? > > Thanks, > > Derry > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Derry\_O\_Sullivan](https://avatars.discourse-cdn.com/v4/letter/d/7bcc69/32.png) [@Derry\_O\_Sullivan](https://discuss.elastic.co/u/Derry_O_Sullivan)\ **Post date:** [September 19, 2013, 7:49am UTC](https://discuss.elastic.co/t/data-cluster-migration-in-seperate-networks/13672/3 "2013-09-19T07:49:55Z")
Hi Alex, Thanks for the response. I guess the reason i want to do this is that i want a way of moving the data from my old cluster to a new cluster with (preferably) no downtime. If i had a good network link, i could think about sharing the data among the networks if testing proved it was ok but i really want to know if ES's replication model will deal with migrating the data automatically and if anyone has used this approach before. Derry On Wednesday, 18 September 2013 22:50:33 UTC+1, Alexander Reelsen wrote: > Hey, > > have you seen the scan search type at > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/search/search-type/) > > is there anything speaking against that in your setup? Just wondering why > you want to have the complexity of connecting clusters across the internet > (with all the stability issues the internet offers). > > --Alex > > On Wed, Sep 18, 2013 at 4:55 PM, Derry O' Sullivan \<[der...@gmail.com](mailto:der...@gmail.com)\ > > > wrote: > > > Hi all, > > > > I'm working on some scalability planning at the moment for our ES > > deployment and looking at a hypothetical situation where we want to move > > our es data from one cluster (e.g. private network or cloud > > (ec2/rackspace/other)) to another. > > > > If i wanted to migrate from this private cluster (e.g. in my own office > > subnet) to e.g. EC2 cluster, is it possible to do 'auto' migration (e.g. > > set up unicast discovery on the ec2 cluster to 'find' the older instance on > > my private network and allow the replica's to be created on EC2). e.g. let > > ES manage the migration. I understand that i may need to increase the > > number of replica's and make sure i have enough nodes in the new cluster to > > meet that requirement > > > > I understand that this option will have latency issues (i.e. would need > > to keep the servers physically close/etc) but would there be any other > > issues? Once the data has been replicated successfully (i.e. enough > > replica's exist on new cluster), the old server cluster could then be > > powered down with the data having been migrated? > > > > Thanks, > > > > Derry > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Norberto\_Meijome](https://avatars.discourse-cdn.com/v4/letter/n/ed655f/32.png) [@Norberto\_Meijome](https://discuss.elastic.co/u/Norberto_Meijome)\ **Post date:** [September 19, 2013, 9:37am UTC](https://discuss.elastic.co/t/data-cluster-migration-in-seperate-networks/13672/4 "2013-09-19T09:37:26Z")
Hey Derry, So I haven't tried this across high latent links, caveat emptor. Assuming you have sorted out your ec2 to internal network access/routing/addressing, unicast should work. You should look into parameters related to timings (connect, discovery, etc) ( sorry, don't have the full docs handy right now). I would hope you have lots of shards , smaller chunks would make for faster distribution I think. Start new nodes in ec2. Something to try would be whether having a copy a snapshot of the data ready for use , i.e., transferred to new nodes outside of ES ( should help to bring online cold shards , if any.) Then yes, increase #replicas. I haven't found that to be super fast.... You can (should) tell your cluster to remove from routing specific old nodes (hint, for multiple nodes, u pass a list of nodes /IPS). Obviously you have to keep an eye on your performance... I don't think there is a one-size- fits all answer. Don't forget to have your load balancer include the new nodes if that's how you have it structured... Good luck, B Hi Alex, Thanks for the response. I guess the reason i want to do this is that i want a way of moving the data from my old cluster to a new cluster with (preferably) no downtime. If i had a good network link, i could think about sharing the data among the networks if testing proved it was ok but i really want to know if ES's replication model will deal with migrating the data automatically and if anyone has used this approach before. Derry On Wednesday, 18 September 2013 22:50:33 UTC+1, Alexander Reelsen wrote: > Hey, > > have you seen the scan search type at > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/search/search-type/) > > is there anything speaking against that in your setup? Just wondering why > you want to have the complexity of connecting clusters across the internet > (with all the stability issues the internet offers). > > --Alex Hi Alex, Thanks for the response. I guess the reason i want to do this is that i want a way of moving the data from my old cluster to a new cluster with (preferably) no downtime. If i had a good network link, i could think about sharing the data among the networks if testing proved it was ok but i really want to know if ES's replication model will deal with migrating the data automatically and if anyone has used this approach before. Derry On Wednesday, 18 September 2013 22:50:33 UTC+1, Alexander Reelsen wrote: > Hey, > > have you seen the scan search type at [http://www.elasticsearch](http://www.elasticsearch).\*\* > org/guide/reference/api/\*\*search/search-type/[http://www.elasticsearch.org/guide/reference/api/search/search-type/](http://www.elasticsearch.org/guide/reference/api/search/search-type/) > > is there anything speaking against that in your setup? Just wondering why > you want to have the complexity of connecting clusters across the internet > (with all the stability issues the internet offers). > > --Alex > > On Wed, Sep 18, 2013 at 4:55 PM, Derry O' Sullivan [der...@gmail.com](mailto:der...@gmail.com)wrote: > > > Hi all, > > > > I'm working on some scalability planning at the moment for our ES > > deployment and looking at a hypothetical situation where we want to move > > our es data from one cluster (e.g. private network or cloud > > (ec2/rackspace/other)) to another. > > > > If i wanted to migrate from this private cluster (e.g. in my own office > > subnet) to e.g. EC2 cluster, is it possible to do 'auto' migration (e.g. > > set up unicast discovery on the ec2 cluster to 'find' the older instance on > > my private network and allow the replica's to be created on EC2). e.g. let > > ES manage the migration. I understand that i may need to increase the > > number of replica's and make sure i have enough nodes in the new cluster to > > meet that requirement > > > > I understand that this option will have latency issues (i.e. would need > > to keep the servers physically close/etc) but would there be any other > > issues? Once the data has been replicated successfully (i.e. enough > > replica's exist on new cluster), the old server cluster could then be > > powered down with the data having been migrated? > > > > Thanks, > > > > Derry > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > . > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:15am UTC](https://discuss.elastic.co/t/data-cluster-migration-in-seperate-networks/13672/5 "2017-07-06T02:15:33Z")