# Cluster Migration Strategies **URL:** \ **Category:** Elasticsearch\ **Created:** [February 28, 2011, 12:23pm UTC](https://discuss.elastic.co/t/cluster-migration-strategies/3995 "2011-02-28T12:23:42Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Paul\_Loy](https://avatars.discourse-cdn.com/v4/letter/p/ad7895/32.png) [@Paul\_Loy](https://discuss.elastic.co/u/Paul_Loy)\ **Post date:** [February 28, 2011, 12:23pm UTC](https://discuss.elastic.co/t/cluster-migration-strategies/3995/1 "2011-02-28T12:23:42Z")
Hi Shay, we're getting close to a production release with elastic search as one of the components in our system. From time-to-time we will want to update ES and also possibly redistribute (change the sharding / replication) the cluster. We think the safest way to do this is probably to start another cluster and migrate the data over. What do you think the best (quickest?) option would be: 1. read from the gateway (is this possible for redistribution?) 2. pull the data from the existing cluster into the new cluster using a) elastic search River? b) custom script / app to pull data across Cheers, Paul ## -- Paul Loy [paul@keteracel.com](mailto:paul@keteracel.com) [http://uk.linkedin.com/in/paulloy](http://uk.linkedin.com/in/paulloy) ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [February 28, 2011, 7:11pm UTC](https://discuss.elastic.co/t/cluster-migration-strategies/3995/2 "2011-02-28T19:11:49Z")
Heya, If the change requires reindexing, then you need to get the docs and index them. The best way is to just get them from the original place where they are stored, or, in upcoming 0.16, use the scan API. If your change does not require reindexing (no mappings were changed / no need to change the number of shards), then you can simply copy over the gateway data (if you using shared gateway), and copy over to each node the "data" directory of existing nodes (this also applied when using the "local" gateway). Make sense? -shay.banon On Monday, February 28, 2011 at 2:23 PM, Paul Loy wrote: > Hi Shay, > > we're getting close to a production release with Elasticsearch as one of the components in our system. From time-to-time we will want to update ES and also possibly redistribute (change the sharding / replication) the cluster. We think the safest way to do this is probably to start another cluster and migrate the data over. What do you think the best (quickest?) option would be: > > 1. read from the gateway (is this possible for redistribution?) > 2. pull the data from the existing cluster into the new cluster using > a) Elasticsearch River? > b) custom script / app to pull data across > > Cheers, > > Paul > > ## -- > > Paul Loy > [paul@keteracel.com](mailto:paul@keteracel.com) > [http://uk.linkedin.com/in/paulloy](http://uk.linkedin.com/in/paulloy) ---
**Author:** ![ppearcy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppearcy/32/980_2.png) [@ppearcy](https://discuss.elastic.co/u/ppearcy)\ **Post date:** [March 1, 2011, 4:20pm UTC](https://discuss.elastic.co/t/cluster-migration-strategies/3995/3 "2011-03-01T16:20:51Z")
Hi, I just tried a similar technique on 0.14.2, but it doesn't appear to be working as I'd expect. I've tried this a couple of times and have gotten exact same results. - Have a three node cluster with local gateway - Updated replica count to 2, so that every machine will have all the shards - Stopped a node, which should now have a full copy of the index data - I then try to rename the cluster on this node by renaming the data directory and updating the elasticsearch.yml file to a new cluster name. I also tweak the config file to only expect one node - I start things up and it appears that a few empty indexes get picked up correctly, but nothing with data loads. The new cluster just sits there in a red state. I bumped up logging to ALL and captured this log: [http://dl.dropbox.com/u/12095883/dev-0.14.2.log](http://dl.dropbox.com/u/12095883/dev-0.14.2.log) Any ideas if I am doing something wrong or hitting a 14.2 issue? Thanks, Paul On Feb 28, 12:11 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > Heya, > > If the change requires reindexing, then you need to get the docs and index them. The best way is to just get them from the original place where they are stored, or, in upcoming 0.16, use the scan API. > > If your change does not require reindexing (no mappings were changed / no need to change the number of shards), then you can simply copy over the gateway data (if you using shared gateway), and copy over to each node the "data" directory of existing nodes (this also applied when using the "local" gateway). > > Make sense? > -shay.banon > > On Monday, February 28, 2011 at 2:23 PM, Paul Loy wrote: > > > Hi Shay, > > > we're getting close to a production release with Elasticsearch as one of the components in our system. From time-to-time we will want to update ES and also possibly redistribute (change the sharding / replication) the cluster. We think the safest way to do this is probably to start another cluster and migrate the data over. What do you think the best (quickest?) option would be: > > > 1. read from the gateway (is this possible for redistribution?) > > 2. pull the data from the existing cluster into the new cluster using > > a) Elasticsearch River? > > b) custom script / app to pull data across > > > Cheers, > > > Paul > > > ## -- > > > > Paul Loy > > [p...@keteracel.com](mailto:p...@keteracel.com) > > [http://uk.linkedin.com/in/paulloy](http://uk.linkedin.com/in/paulloy) ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [March 1, 2011, 4:47pm UTC](https://discuss.elastic.co/t/cluster-migration-strategies/3995/4 "2011-03-01T16:47:19Z")
Hi Paul > - I then try to rename the cluster on this node by renaming the data > directory and updating the elasticsearch.yml file to a new cluster > name. I also tweak the config file to only expect one node Could this be something to do with this bug: > > > The cluster name is no longer honored via the TransportClient unless sniffing is… enable don 0.14.2. See this discussion for more details: > http://elasticsearch-users.115913.n3.nabble.com/0-14-2-TransportClient-does-not-appear-to-be-honoring-cluster-name-tp2289073p2289073.html > > Thanks! [http://elasticsearch-users.115913.n3.nabble.com/0-14-2-TransportClient-does-not-appear-to-be-honoring-cluster-name-td2289073.html](http://elasticsearch-users.115913.n3.nabble.com/0-14-2-TransportClient-does-not-appear-to-be-honoring-cluster-name-td2289073.html) So your existing 0.14.2 nodes are confusing the new node, with the new cluster name? You could try setting the new node to use different ports, so that there is no chance that the new and old nodes could talk clint ---
**Author:** ![ppearcy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppearcy/32/980_2.png) [@ppearcy](https://discuss.elastic.co/u/ppearcy)\ **Post date:** [March 1, 2011, 5:33pm UTC](https://discuss.elastic.co/t/cluster-migration-strategies/3995/5 "2011-03-01T17:33:28Z")
Thanks Clinton. I didn't see any indication that there is any cross communication between old and new clusters, but tried changing the port and am getting the same behavior. On Mar 1, 9:47 am, Clinton Gormley [clin...@iannounce.co.uk](mailto:clin...@iannounce.co.uk) wrote: > Hi Paul > > > - I then try to rename the cluster on this node by renaming the data > > directory and updating the elasticsearch.yml file to a new cluster > > name. I also tweak the config file to only expect one node > > Could this be something to do with this bug:[https://github.com/elasticsearch/elasticsearch/issues/637http://elasticsearch-users.115913.n3.nabble.com/0-14-2-TransportClien](https://github.com/elasticsearch/elasticsearch/issues/637http://elasticsearch-users.115913.n3.nabble.com/0-14-2-TransportClien)... > > So your existing 0.14.2 nodes are confusing the new node, with the new > cluster name? > > You could try setting the new node to use different ports, so that there > is no chance that the new and old nodes could talk > > clint ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [March 2, 2011, 12:39am UTC](https://discuss.elastic.co/t/cluster-migration-strategies/3995/6 "2011-03-02T00:39:53Z")
No data will be loaded because with 2 replicas, the cluster will recover an index once 2 instances of a shard are found within the cluster. You don't have to increase the number of replicas, you can simply copy over the 3 nodes data directory to the new 3 machine, rename the cluster name (and rename it under hte data dir) and it should work. On Tuesday, March 1, 2011 at 7:33 PM, Paul wrote: > Thanks Clinton. > > I didn't see any indication that there is any cross communication > between old and new clusters, but tried changing the port and am > getting the same behavior. > > On Mar 1, 9:47 am, Clinton Gormley [clin...@iannounce.co.uk](mailto:clin...@iannounce.co.uk) wrote: > > > Hi Paul > > > > > - I then try to rename the cluster on this node by renaming the data > > > directory and updating the elasticsearch.yml file to a new cluster > > > name. I also tweak the config file to only expect one node > > > > Could this be something to do with this bug:[https://github.com/elasticsearch/elasticsearch/issues/637http://elasticsearch-users.115913.n3.nabble.com/0-14-2-TransportClien](https://github.com/elasticsearch/elasticsearch/issues/637http://elasticsearch-users.115913.n3.nabble.com/0-14-2-TransportClien)... > > > > So your existing 0.14.2 nodes are confusing the new node, with the new > > cluster name? > > > > You could try setting the new node to use different ports, so that there > > is no chance that the new and old nodes could talk > > > > clint ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:11am UTC](https://discuss.elastic.co/t/cluster-migration-strategies/3995/7 "2017-07-06T04:11:07Z")