# Migrating to new cluster

**URL:** <https://discuss.elastic.co/t/migrating-to-new-cluster/15898>\
**Category:** Elasticsearch\
**Created:** [February 19, 2014, 3:01pm UTC](https://discuss.elastic.co/t/migrating-to-new-cluster/15898 "2014-02-19T15:01:03Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![Attila\_Bukor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/attila_bukor/32/1667_2.png) [@Attila\_Bukor](https://discuss.elastic.co/u/Attila_Bukor)\
**Post date:** [February 19, 2014, 3:01pm UTC](https://discuss.elastic.co/t/migrating-to-new-cluster/15898/1 "2014-02-19T15:01:03Z")

</div>

Hey everybody,

I have a question regarding the migration of the indices to a new cluster  
seamlessly. Let me first describe the situation:

I have a project which uses Elasticsearch since a few weeks ago. This is our  
first Elasticsearch project, and as we are satisfied with it, we decided to  
dedicate a cluster of 3 nodes to run Elasticsearch (lets call it  
"newcluster"). It is up and running separately from our old 1-node  
Elasticsearch installation ("oldcluster"). We want to migrate everything to  
newcluster, and turn oldcluster off. All 4 servers are running Ubuntu Server,  
Elasticsearch is installed from the official deb package, oldcluster is  
0.90.10 and newcluster 1.0.0.

I came up with the following solution, but I'm not sure it will work:

- Upgrade oldcluster to 1.0.0 (tested it in dev, should work)
- Backup the index (we have only one index right now) in case something goes  
wrong.
- Change oldcluster's name to newcluster.
- Wait until the nodes synchronize themselves (how can I check if they're in  
sync?)
- Shut down oldcluster.

Thank you very much,  
Attila

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/alpine.LNX.2.03.1402191600190.1091%40kara](https://groups.google.com/d/msgid/elasticsearch/alpine.LNX.2.03.1402191600190.1091%40kara).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Yann\_Barraud](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yann_barraud/32/1108_2.png) [@Yann\_Barraud](https://discuss.elastic.co/u/Yann_Barraud)\
**Post date:** [February 19, 2014, 3:14pm UTC](https://discuss.elastic.co/t/migrating-to-new-cluster/15898/2 "2014-02-19T15:14:43Z")

</div>

Hi,

If I get you well, you'll face an issue. Renaming oldcluster will make it  
"forget" previous indexes. (Take a look at data you'll find a dir named  
"oldcluster").

The step you're missing is to backup data fro oldcluster  
Then shtudown, rename  
Then restore...

Or even shutdown and restore on new cluster...

Cheers,  
Yann

Le mercredi 19 février 2014 16:01:03 UTC+1, Attila Bukor a écrit :

> Hey everybody,
> 
> I have a question regarding the migration of the indices to a new cluster  
> seamlessly. Let me first describe the situation:
> 
> I have a project which uses Elasticsearch since a few weeks ago. This is  
> our  
> first Elasticsearch project, and as we are satisfied with it, we decided  
> to  
> dedicate a cluster of 3 nodes to run Elasticsearch (lets call it  
> "newcluster"). It is up and running separately from our old 1-node  
> Elasticsearch installation ("oldcluster"). We want to migrate everything  
> to  
> newcluster, and turn oldcluster off. All 4 servers are running Ubuntu  
> Server,  
> Elasticsearch is installed from the official deb package, oldcluster is  
> 0.90.10 and newcluster 1.0.0.
> 
> I came up with the following solution, but I'm not sure it will work:
> 
> - Upgrade oldcluster to 1.0.0 (tested it in dev, should work)
> - Backup the index (we have only one index right now) in case something  
> goes  
> wrong.
> - Change oldcluster's name to newcluster.
> - Wait until the nodes synchronize themselves (how can I check if they're  
> in  
> sync?)
> - Shut down oldcluster.
> 
> Thank you very much,  
> Attila

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/dc031e6c-a7d8-4773-a15d-293986331c10%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/dc031e6c-a7d8-4773-a15d-293986331c10%40googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Tony\_Su](https://avatars.discourse-cdn.com/v4/letter/t/8dc957/32.png) [@Tony\_Su](https://discuss.elastic.co/u/Tony_Su)\
**Post date:** [February 19, 2014, 3:50pm UTC](https://discuss.elastic.co/t/migrating-to-new-cluster/15898/3 "2014-02-19T15:50:05Z")

</div>

Hi,  
My testing has done a little bit of what you're describing...

- FWIW and I don't know why, when I re-insert the data from scratch instead  
of upgrading a cluster, I'm seeing substantial improvements in node general  
health. They seem faster, less latencies related to indexing (I assume  
there should be no diff searching, but ??). If a reason why I'm seeing this  
can't be found, then maybe what I'm seeing is specific to my setup and  
hardware.

- I don't know that there should be any reason to name your cluster  
differently if the networks are completely isolated, but of course if there  
should be a mistake, the consequences could be serious.

- I've been using various methods that display index activity, the easiest  
to use (and today work on both 0.90 and 1.0) are elasticsearch-head and  
elasticsearch-hq. I've also been toying around recently with simply probing  
each node using a curl command, which depending on how you deploy web tools  
likely has a lower load effect on resources (I'm currently running Apache  
on one node and when I use one or more of the above tools, has a  
significant and measurable load).

- If you do an upgrade, at least the one time after the upgrade you may  
want to force your cluster to wait until all nodes in the cluster are  
active before restarting to minimize shard thrashing (if a node is not  
online yet, the cluster may discard the indices on that node and  
re-allocate from whatever copy already is active. Then, when the missing  
node finally joins, then the shards are re-balanced). One way to do that is  
setting the following to be the same as the total nodes in the cluster

gateway.recovery\_after\_nodes

HTH,  
Tony

On Wednesday, February 19, 2014 7:01:03 AM UTC-8, Attila Bukor wrote:

> Hey everybody,
> 
> I have a question regarding the migration of the indices to a new cluster  
> seamlessly. Let me first describe the situation:
> 
> I have a project which uses Elasticsearch since a few weeks ago. This is  
> our  
> first Elasticsearch project, and as we are satisfied with it, we decided  
> to  
> dedicate a cluster of 3 nodes to run Elasticsearch (lets call it  
> "newcluster"). It is up and running separately from our old 1-node  
> Elasticsearch installation ("oldcluster"). We want to migrate everything  
> to  
> newcluster, and turn oldcluster off. All 4 servers are running Ubuntu  
> Server,  
> Elasticsearch is installed from the official deb package, oldcluster is  
> 0.90.10 and newcluster 1.0.0.
> 
> I came up with the following solution, but I'm not sure it will work:
> 
> - Upgrade oldcluster to 1.0.0 (tested it in dev, should work)
> - Backup the index (we have only one index right now) in case something  
> goes  
> wrong.
> - Change oldcluster's name to newcluster.
> - Wait until the nodes synchronize themselves (how can I check if they're  
> in  
> sync?)
> - Shut down oldcluster.
> 
> Thank you very much,  
> Attila

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5a1c622f-4ac5-4203-ab00-d66facb227a9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5a1c622f-4ac5-4203-ab00-d66facb227a9%40googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Attila\_Bukor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/attila_bukor/32/1667_2.png) [@Attila\_Bukor](https://discuss.elastic.co/u/Attila_Bukor)\
**Post date:** [February 19, 2014, 4:16pm UTC](https://discuss.elastic.co/t/migrating-to-new-cluster/15898/4 "2014-02-19T16:16:02Z")

</div>

Hi Yann,

Thank you for your response, you saved me from a _lot_ of headache. So the  
revised flow:

- Upgrade oldcluster to 1.0.0.
- Implement a logger in the application which dumps all index API calls  
into a file.
- Backup my index on oldcluster following these docs:

> **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)**
>
> Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today.

- Restore the backed up index on newcluster following the same docs.
- Run the log from after the last \_id in the snapshot. Repeat until  
they're the same.
- Change the IP to newcluster in the application.
- Repeat step 5 one more time.
- Remove logging, delete logs.
- Shut down oldcluster.

In this case, the switch should not be noticed by anyone and the data  
integrity is guaranteed, right?

Cheers,  
Attila

On Wed, 19 Feb 2014, Yann Barraud wrote:

> Hi,
> 
> If I get you well, you'll face an issue. Renaming oldcluster will  
> make it "forget" previous indexes. (Take a look at data you'll find a  
> dir named "oldcluster").
> 
> The step you're missing is to backup data fro oldcluster  
> Then shtudown, rename  
> Then restore...
> 
> Or even shutdown and restore on new cluster...
> 
> Cheers,  
> Yann
> 
> Le mercredi 19 février 2014 16:01:03 UTC+1, Attila Bukor a écrit :  
> Hey everybody,
> 
> ```
> I have a question regarding the migration of the indices to a
> 
> ```

new cluster

> ```
> seamlessly. Let me first describe the situation:
> 
> I have a project which uses Elasticsearch since a few weeks
> 
> ```

ago. This is our

> ```
> first Elasticsearch project, and as we are satisfied with it,
> 
> ```

we decided to

> ```
> dedicate a cluster of 3 nodes to run Elasticsearch (lets call it
> "newcluster"). It is up and running separately from our old 1-node
> Elasticsearch installation ("oldcluster"). We want to migrate
> 
> ```

everything to

> ```
> newcluster, and turn oldcluster off. All 4 servers are running
> 
> ```

Ubuntu Server,

> ```
> Elasticsearch is installed from the official deb package,
> 
> ```

oldcluster is

> ```
> 0.90.10 and newcluster 1.0.0.
> 
> I came up with the following solution, but I'm not sure it will
> 
> ```

work:

> ```
> - Upgrade oldcluster to 1.0.0 (tested it in dev, should work)
> - Backup the index (we have only one index right now) in case
> 
> ```

something goes

> ```
> wrong.
> - Change oldcluster's name to newcluster.
> - Wait until the nodes synchronize themselves (how can I check
> 
> ```

if they're in

> ```
> sync?)
> - Shut down oldcluster.
> 
> Thank you very much,
> Attila
> 
> ```

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/le2lbk%24v8v%241%40ger.gmane.org](https://groups.google.com/d/msgid/elasticsearch/le2lbk%24v8v%241%40ger.gmane.org).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Attila\_Bukor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/attila_bukor/32/1667_2.png) [@Attila\_Bukor](https://discuss.elastic.co/u/Attila_Bukor)\
**Post date:** [February 19, 2014, 4:26pm UTC](https://discuss.elastic.co/t/migrating-to-new-cluster/15898/5 "2014-02-19T16:26:15Z")

</div>

Hi Tony,

Thank you for your response, reading this I'm pretty sure I'm going with  
Yann's suggestion to simply backup & restore. Your first point confirms that  
this is a good idea anyway.

To answer your second point, both clusters are on the same /24 subnet, the  
problem is that oldcluster is not designed to do that, we put a production  
site's index on a server with twenty-odd GB free space and 4GB RAM total, so  
we _need_ to move it to the dedicated cluster.

The other two points are great suggestions and thank you very much for them,  
but I think the backup & restore way would be more suitable for my case.

Thank you very much,  
Attila

On 02/19/2014 04:50 PM, Tony Su wrote:\> Hi,

> My testing has done a little bit of what you're describing...
> 
> - FWIW and I don't know why, when I re-insert the data from scratch  
> instead of upgrading a cluster, I'm seeing substantial improvements in  
> node general health. They seem faster, less latencies related to  
> indexing (I assume there should be no diff searching, but ??). If a  
> reason why I'm seeing this can't be found, then maybe what I'm seeing is  
> specific to my setup and hardware.
> - I don't know that there should be any reason to name your cluster  
> differently if the networks are completely isolated, but of course if  
> there should be a mistake, the consequences could be serious.
> - I've been using various methods that display index activity, the  
> easiest to use (and today work on both 0.90 and 1.0) are  
> elasticsearch-head and elasticsearch-hq. I've also been toying around  
> recently with simply probing each node using a curl command, which  
> depending on how you deploy web tools likely has a lower load effect on  
> resources (I'm currently running Apache on one node and when I use one  
> or more of the above tools, has a significant and measurable load).
> - If you do an upgrade, at least the one time after the upgrade you may  
> want to force your cluster to wait until all nodes in the cluster are  
> active before restarting to minimize shard thrashing (if a node is not  
> online yet, the cluster may discard the indices on that node and  
> re-allocate from whatever copy already is active. Then, when the missing  
> node finally joins, then the shards are re-balanced). One way to do that  
> is setting the following to be the same as the total nodes in the cluster  
> gateway.recovery\_after\_nodes  
> HTH,  
> Tony
> 
> On Wednesday, February 19, 2014 7:01:03 AM UTC-8, Attila Bukor wrote:
> 
> ```
> Hey everybody,
> 
> I have a question regarding the migration of the indices to a new
> cluster
> seamlessly. Let me first describe the situation:
> 
> I have a project which uses Elasticsearch since a few weeks ago.
> This is our
> first Elasticsearch project, and as we are satisfied with it, we
> decided to
> dedicate a cluster of 3 nodes to run Elasticsearch (lets call it
> "newcluster"). It is up and running separately from our old 1-node
> Elasticsearch installation ("oldcluster"). We want to migrate
> everything to
> newcluster, and turn oldcluster off. All 4 servers are running
> Ubuntu Server,
> Elasticsearch is installed from the official deb package, 
> 
> ```

oldcluster is

> ```
> 0.90.10 and newcluster 1.0.0.
> 
> I came up with the following solution, but I'm not sure it will work:
> 
> - Upgrade oldcluster to 1.0.0 (tested it in dev, should work)
> - Backup the index (we have only one index right now) in case
> something goes
> wrong.
> - Change oldcluster's name to newcluster.
> - Wait until the nodes synchronize themselves (how can I check if
> they're in
> sync?)
> - Shut down oldcluster.
> 
> Thank you very much,
> Attila
> 
> ```
> 
> --  
> You received this message because you are subscribed to the Google  
> Groups "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send  
> an email to

[elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).

> To view this discussion on the web visit

[https://groups.google.com/d/msgid/elasticsearch/5a1c622f-4ac5-4203-ab00-d66facb227a9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5a1c622f-4ac5-4203-ab00-d66facb227a9%40googlegroups.com).

> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/le2luq%249t8%241%40ger.gmane.org](https://groups.google.com/d/msgid/elasticsearch/le2luq%249t8%241%40ger.gmane.org).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Tony\_Su](https://avatars.discourse-cdn.com/v4/letter/t/8dc957/32.png) [@Tony\_Su](https://discuss.elastic.co/u/Tony_Su)\
**Post date:** [February 19, 2014, 6:00pm UTC](https://discuss.elastic.co/t/migrating-to-new-cluster/15898/6 "2014-02-19T18:00:17Z")

</div>

I was wondering which backup/restore method you intended to use.

Snapshot/restore requires 1.0.  
If you intend to upgrade your 0.9.x cluster to 1.0 first, it looks like  
possibly a really good option (I intend to start testing snapshot/restore  
sometime soon. I've already created snapshots and am impressed how it seems  
to take roughly the same amount to create a snapshot no matter how much  
data is being backed up, probably within reason)

If your chosen path requires upgrading the existing cluster to 1.0 anyway,  
why migrate to the new cluster?

Tony

On Wednesday, February 19, 2014 8:16:02 AM UTC-8, Attila Bukor wrote:

> Hi Yann,
> 
> Thank you for your response, you saved me from a _lot_ of headache. So the  
> revised flow:
> 
> - Upgrade oldcluster to 1.0.0.
> - Implement a logger in the application which dumps all index API calls  
> into a file.
> - Backup my index on oldcluster following these docs:
> 
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-snapshots.html)
> 
> - Restore the backed up index on newcluster following the same docs.
> - Run the log from after the last \_id in the snapshot. Repeat until  
> they're the same.
> - Change the IP to newcluster in the application.
> - Repeat step 5 one more time.
> - Remove logging, delete logs.
> - Shut down oldcluster.
> 
> In this case, the switch should not be noticed by anyone and the data  
> integrity is guaranteed, right?
> 
> Cheers,  
> Attila
> 
> On Wed, 19 Feb 2014, Yann Barraud wrote:
> 
> > Hi,
> > 
> > If I get you well, you'll face an issue. Renaming oldcluster will  
> > make it "forget" previous indexes. (Take a look at data you'll find a  
> > dir named "oldcluster").
> > 
> > The step you're missing is to backup data fro oldcluster  
> > Then shtudown, rename  
> > Then restore...
> > 
> > Or even shutdown and restore on new cluster...
> > 
> > Cheers,  
> > Yann
> > 
> > Le mercredi 19 février 2014 16:01:03 UTC+1, Attila Bukor a écrit :  
> > Hey everybody,
> > 
> > ```
> > I have a question regarding the migration of the indices to a 
> > 
> > ```
> 
> new cluster
> 
> > ```
> > seamlessly. Let me first describe the situation: 
> > 
> > I have a project which uses Elasticsearch since a few weeks 
> > 
> > ```
> 
> ago. This is our
> 
> > ```
> > first Elasticsearch project, and as we are satisfied with it, 
> > 
> > ```
> 
> we decided to
> 
> > ```
> > dedicate a cluster of 3 nodes to run Elasticsearch (lets call it 
> > "newcluster"). It is up and running separately from our old 1-node 
> > Elasticsearch installation ("oldcluster"). We want to migrate 
> > 
> > ```
> 
> everything to
> 
> > ```
> > newcluster, and turn oldcluster off. All 4 servers are running 
> > 
> > ```
> 
> Ubuntu Server,
> 
> > ```
> > Elasticsearch is installed from the official deb package, 
> > 
> > ```
> 
> oldcluster is
> 
> > ```
> > 0.90.10 and newcluster 1.0.0. 
> > 
> > I came up with the following solution, but I'm not sure it will 
> > 
> > ```
> 
> work:
> 
> > ```
> > - Upgrade oldcluster to 1.0.0 (tested it in dev, should work) 
> > - Backup the index (we have only one index right now) in case 
> > 
> > ```
> 
> something goes
> 
> > ```
> > wrong. 
> > - Change oldcluster's name to newcluster. 
> > - Wait until the nodes synchronize themselves (how can I check 
> > 
> > ```
> 
> if they're in
> 
> > ```
> > sync?) 
> > - Shut down oldcluster. 
> > 
> > Thank you very much, 
> > Attila 
> > 
> > ```

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/f375faef-2dd4-416d-8150-75afe2318f0c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f375faef-2dd4-416d-8150-75afe2318f0c%40googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Attila\_Bukor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/attila_bukor/32/1667_2.png) [@Attila\_Bukor](https://discuss.elastic.co/u/Attila_Bukor)\
**Post date:** [February 20, 2014, 11:03am UTC](https://discuss.elastic.co/t/migrating-to-new-cluster/15898/7 "2014-02-20T11:03:44Z")

</div>

Hi Tony,

The problem is the hardware, that is why we need to move it to the new  
cluster.

Yann, how can I restore a snapshot on a different machine?

Cheers,  
Attila

On 02/19/2014 07:00 PM, Tony Su wrote:

> I was wondering which backup/restore method you intended to use.
> 
> Snapshot/restore requires 1.0.  
> If you intend to upgrade your 0.9.x cluster to 1.0 first, it looks like  
> possibly a really good option (I intend to start testing  
> snapshot/restore sometime soon. I've already created snapshots and am  
> impressed how it seems to take roughly the same amount to create a  
> snapshot no matter how much data is being backed up, probably within reason)
> 
> If your chosen path requires upgrading the existing cluster to 1.0  
> anyway, why migrate to the new cluster?
> 
> Tony
> 
> On Wednesday, February 19, 2014 8:16:02 AM UTC-8, Attila Bukor wrote:
> 
> ```
> Hi Yann,
> 
> Thank you for your response, you saved me from a *lot* of headache.
> So the
> revised flow:
> 
> - Upgrade oldcluster to 1.0.0.
> - Implement a logger in the application which dumps all index API calls
> into a file.
> - Backup my index on oldcluster following these docs:
> 
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-snapshots.html
> <http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-snapshots.html>
> 
> - Restore the backed up index on newcluster following the same docs.
> - Run the log from after the last _id in the snapshot. Repeat until
> they're the same.
> - Change the IP to newcluster in the application.
> - Repeat step 5 one more time.
> - Remove logging, delete logs.
> - Shut down oldcluster.
> 
> In this case, the switch should not be noticed by anyone and the data
> integrity is guaranteed, right?
> 
> Cheers,
> Attila
> 
> On Wed, 19 Feb 2014, Yann Barraud wrote:
> 
> > Hi,
> >
> > If I get you well, you'll face an issue. Renaming oldcluster will
> make it "forget" previous indexes. (Take a look at data you'll find a
> dir named "oldcluster").
> >
> > The step you're missing is to backup data fro oldcluster
> > Then shtudown, rename
> > Then restore...
> >
> > Or even shutdown and restore on new cluster...
> >
> > Cheers,
> > Yann
> >
> >
> >
> > Le mercredi 19 février 2014 16:01:03 UTC+1, Attila Bukor a écrit :
> > Hey everybody,
> >
> > I have a question regarding the migration of the indices to a
> new cluster
> > seamlessly. Let me first describe the situation:
> >
> > I have a project which uses Elasticsearch since a few weeks
> ago. This is our
> > first Elasticsearch project, and as we are satisfied with it,
> we decided to
> > dedicate a cluster of 3 nodes to run Elasticsearch (lets
> call it
> > "newcluster"). It is up and running separately from our old
> 1-node
> > Elasticsearch installation ("oldcluster"). We want to migrate
> everything to
> > newcluster, and turn oldcluster off. All 4 servers are running
> Ubuntu Server,
> > Elasticsearch is installed from the official deb package,
> oldcluster is
> > 0.90.10 and newcluster 1.0.0.
> >
> > I came up with the following solution, but I'm not sure it
> will
> work:
> >
> > - Upgrade oldcluster to 1.0.0 (tested it in dev, should work)
> > - Backup the index (we have only one index right now) in case
> something goes
> > wrong.
> > - Change oldcluster's name to newcluster.
> > - Wait until the nodes synchronize themselves (how can I check
> if they're in
> > sync?)
> > - Shut down oldcluster.
> >
> > Thank you very much,
> > Attila
> >
> >
> >
> 
> ```
> 
> --  
> You received this message because you are subscribed to the Google  
> Groups "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send  
> an email to  
> [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/f375faef-2dd4-416d-8150-75afe2318f0c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f375faef-2dd4-416d-8150-75afe2318f0c%40googlegroups.com).  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/le4ne3%24cg5%241%40ger.gmane.org](https://groups.google.com/d/msgid/elasticsearch/le4ne3%24cg5%241%40ger.gmane.org).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Yann\_Barraud](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yann_barraud/32/1108_2.png) [@Yann\_Barraud](https://discuss.elastic.co/u/Yann_Barraud)\
**Post date:** [February 20, 2014, 12:14pm UTC](https://discuss.elastic.co/t/migrating-to-new-cluster/15898/8 "2014-02-20T12:14:09Z")

</div>

[Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html#_restore:-))

Cordialement,  
Yann Barraud

2014-02-20 12:03 GMT+01:00 Attila Bukor [r1pp3rj4ck@w4it.eu](mailto:r1pp3rj4ck@w4it.eu):

> Hi Tony,
> 
> The problem is the hardware, that is why we need to move it to the new  
> cluster.
> 
> Yann, how can I restore a snapshot on a different machine?
> 
> Cheers,  
> Attila
> 
> On 02/19/2014 07:00 PM, Tony Su wrote:
> 
> > I was wondering which backup/restore method you intended to use.
> > 
> > Snapshot/restore requires 1.0.  
> > If you intend to upgrade your 0.9.x cluster to 1.0 first, it looks like  
> > possibly a really good option (I intend to start testing  
> > snapshot/restore sometime soon. I've already created snapshots and am  
> > impressed how it seems to take roughly the same amount to create a  
> > snapshot no matter how much data is being backed up, probably within  
> > reason)
> > 
> > If your chosen path requires upgrading the existing cluster to 1.0  
> > anyway, why migrate to the new cluster?
> > 
> > Tony
> > 
> > On Wednesday, February 19, 2014 8:16:02 AM UTC-8, Attila Bukor wrote:
> > 
> > ```
> > Hi Yann,
> > 
> > Thank you for your response, you saved me from a *lot* of headache.
> > So the
> > revised flow:
> > 
> > - Upgrade oldcluster to 1.0.0.
> > - Implement a logger in the application which dumps all index API
> > 
> > ```
> > 
> > calls  
> > into a file.  
> > - Backup my index on oldcluster following these docs:
> > 
> > ```
> > http://www.elasticsearch.org/guide/en/elasticsearch/
> > 
> > ```
> > 
> > reference/master/modules-snapshots.html  
> > \<[Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/)  
> > reference/master/modules-snapshots.html\>
> > 
> > ```
> > - Restore the backed up index on newcluster following the same docs.
> > - Run the log from after the last _id in the snapshot. Repeat until
> > they're the same.
> > - Change the IP to newcluster in the application.
> > - Repeat step 5 one more time.
> > - Remove logging, delete logs.
> > - Shut down oldcluster.
> > 
> > In this case, the switch should not be noticed by anyone and the data
> > integrity is guaranteed, right?
> > 
> > Cheers,
> > Attila
> > 
> > On Wed, 19 Feb 2014, Yann Barraud wrote:
> > 
> > > Hi,
> > >
> > > If I get you well, you'll face an issue. Renaming oldcluster will
> > make it "forget" previous indexes. (Take a look at data you'll find a
> > dir named "oldcluster").
> > >
> > > The step you're missing is to backup data fro oldcluster
> > > Then shtudown, rename
> > > Then restore...
> > >
> > > Or even shutdown and restore on new cluster...
> > >
> > > Cheers,
> > > Yann
> > >
> > >
> > >
> > > Le mercredi 19 février 2014 16:01:03 UTC+1, Attila Bukor a écrit :
> > > Hey everybody,
> > >
> > > I have a question regarding the migration of the indices to a
> > new cluster
> > > seamlessly. Let me first describe the situation:
> > >
> > > I have a project which uses Elasticsearch since a few weeks
> > ago. This is our
> > > first Elasticsearch project, and as we are satisfied with it,
> > we decided to
> > > dedicate a cluster of 3 nodes to run Elasticsearch (lets
> > call it
> > > "newcluster"). It is up and running separately from our old
> > 1-node
> > > Elasticsearch installation ("oldcluster"). We want to migrate
> > everything to
> > > newcluster, and turn oldcluster off. All 4 servers are
> > 
> > ```
> > 
> > running  
> > Ubuntu Server,  
> > \> Elasticsearch is installed from the official deb package,  
> > oldcluster is  
> > \> 0.90.10 and newcluster 1.0.0.  
> > \>  
> > \> I came up with the following solution, but I'm not sure it  
> > will  
> > work:  
> > \>  
> > \> - Upgrade oldcluster to 1.0.0 (tested it in dev, should work)  
> > \> - Backup the index (we have only one index right now) in case  
> > something goes  
> > \> wrong.  
> > \> - Change oldcluster's name to newcluster.  
> > \> - Wait until the nodes synchronize themselves (how can I  
> > check  
> > if they're in  
> > \> sync?)  
> > \> - Shut down oldcluster.  
> > \>  
> > \> Thank you very much,  
> > \> Attila  
> > \>  
> > \>  
> > \>
> > 
> > --  
> > You received this message because you are subscribed to the Google  
> > Groups "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send
> > 
> > an email to  
> > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> > To view this discussion on the web visit  
> > [https://groups.google.com/d/msgid/elasticsearch/f375faef-](https://groups.google.com/d/msgid/elasticsearch/f375faef-)  
> > 2dd4-416d-8150-75afe2318f0c%[40googlegroups.com](http://40googlegroups.com).  
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> 
> --  
> You received this message because you are subscribed to a topic in the  
> Google Groups "elasticsearch" group.  
> To unsubscribe from this topic, visit [https://groups.google.com/d/](https://groups.google.com/d/)  
> topic/elasticsearch/aOVumOxmsuQ/unsubscribe.  
> To unsubscribe from this group and all its topics, send an email to  
> [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)  
> msgid/elasticsearch/le4ne3%24cg5%241%[40ger.gmane.org](http://40ger.gmane.org).
> 
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CA%2BhvuXc%2BuQWV5ySXa0407rHCmafEef4x4OU0qdBJ5FNLzMy0yQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CA%2BhvuXc%2BuQWV5ySXa0407rHCmafEef4x4OU0qdBJ5FNLzMy0yQ%40mail.gmail.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Attila\_Bukor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/attila_bukor/32/1667_2.png) [@Attila\_Bukor](https://discuss.elastic.co/u/Attila_Bukor)\
**Post date:** [February 20, 2014, 12:42pm UTC](https://discuss.elastic.co/t/migrating-to-new-cluster/15898/9 "2014-02-20T12:42:11Z")

</div>

Hi,

I've read this one, but it's not clear to me how to do it on a different  
machine.

If I do a backup on oldcluster with

curl -XPUT oldcluster:9200/\_snapshot/my\_backup/snapshot\_1 (assuming I've  
already created the my\_backup repo), there will be metadata-snapshot\_1,  
snapshot-snapshot\_1 and some other files under  
/var/lib/elasticsearch/my\_backup or some other place which is configured.

Should I just tar the whole directory, scp it to node1 on newcluster and  
call

curl -XPOST newcluster-node1:9200/\_snapshot/my\_backup/snapshot\_1/restore ?

I don't really want to break things and this situation is not really  
explained in the docs.

Cheers,  
Attila

On 02/20/2014 01:14 PM, Yann Barraud wrote:

> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html#_restore)  
> 🙂
> 
> Cordialement,  
> Yann Barraud
> 
> 2014-02-20 12:03 GMT+01:00 Attila Bukor  
> \<r1pp3rj4ck@w4it.eu  
> [mailto:r1pp3rj4ck@w4it.eu](mailto:r1pp3rj4ck@w4it.eu)\>:
> 
> ```
> Hi Tony,
> 
> The problem is the hardware, that is why we need to move it to the new
> cluster.
> 
> Yann, how can I restore a snapshot on a different machine?
> 
> Cheers,
> Attila
> 
> On 02/19/2014 07:00 PM, Tony Su wrote:
> 
> I was wondering which backup/restore method you intended to use.
> 
> Snapshot/restore requires 1.0.
> If you intend to upgrade your 0.9.x cluster to 1.0 first, it
> looks like
> possibly a really good option (I intend to start testing
> snapshot/restore sometime soon. I've already created snapshots
> and am
> impressed how it seems to take roughly the same amount to create a
> snapshot no matter how much data is being backed up, probably
> within reason)
> 
> If your chosen path requires upgrading the existing cluster to 1.0
> anyway, why migrate to the new cluster?
> 
> Tony
> 
> On Wednesday, February 19, 2014 8:16:02 AM UTC-8, Attila Bukor
> wrote:
> 
> Hi Yann,
> 
> Thank you for your response, you saved me from a *lot* of
> headache.
> So the
> revised flow:
> 
> - Upgrade oldcluster to 1.0.0.
> - Implement a logger in the application which dumps all
> index API calls
> into a file.
> - Backup my index on oldcluster following these docs:
> 
> http://www.elasticsearch.org/ __guide/en/elasticsearch/__ reference/master/modules-__snapshots.html
> <http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-snapshots.html>
> 
> <http://www.elasticsearch.org/ __guide/en/elasticsearch/__ reference/master/modules-__snapshots.html
> <http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-snapshots.html>>
> 
> - Restore the backed up index on newcluster following the
> same docs.
> - Run the log from after the last _id in the snapshot.
> Repeat until
> they're the same.
> - Change the IP to newcluster in the application.
> - Repeat step 5 one more time.
> - Remove logging, delete logs.
> - Shut down oldcluster.
> 
> In this case, the switch should not be noticed by anyone
> and the data
> integrity is guaranteed, right?
> 
> Cheers,
> Attila
> 
> On Wed, 19 Feb 2014, Yann Barraud wrote:
> 
> > Hi,
> >
> > If I get you well, you'll face an issue. Renaming
> oldcluster will
> make it "forget" previous indexes. (Take a look at data
> you'll find a
> dir named "oldcluster").
> >
> > The step you're missing is to backup data fro oldcluster
> > Then shtudown, rename
> > Then restore...
> >
> > Or even shutdown and restore on new cluster...
> >
> > Cheers,
> > Yann
> >
> >
> >
> > Le mercredi 19 février 2014 16:01:03 UTC+1, Attila Bukor
> a écrit :
> > Hey everybody,
> >
> > I have a question regarding the migration of the
> indices to a
> new cluster
> > seamlessly. Let me first describe the situation:
> >
> > I have a project which uses Elasticsearch since a
> few weeks
> ago. This is our
> > first Elasticsearch project, and as we are
> satisfied with it,
> we decided to
> > dedicate a cluster of 3 nodes to run Elasticsearch
> (lets
> call it
> > "newcluster"). It is up and running separately
> from our old
> 1-node
> > Elasticsearch installation ("oldcluster"). We want
> to migrate
> everything to
> > newcluster, and turn oldcluster off. All 4 servers
> are running
> Ubuntu Server,
> > Elasticsearch is installed from the official deb
> package,
> oldcluster is
> > 0.90.10 and newcluster 1.0.0.
> >
> > I came up with the following solution, but I'm not
> sure it
> will
> work:
> >
> > - Upgrade oldcluster to 1.0.0 (tested it in dev,
> should work)
> > - Backup the index (we have only one index right
> now) in case
> something goes
> > wrong.
> > - Change oldcluster's name to newcluster.
> > - Wait until the nodes synchronize themselves (how
> can I check
> if they're in
> > sync?)
> > - Shut down oldcluster.
> >
> > Thank you very much,
> > Attila
> >
> >
> >
> 
> --
> You received this message because you are subscribed to the Google
> Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from
> it, send
> 
> an email to
> elasticsearch+unsubscribe@__googlegroups.com
> <mailto:elasticsearch%2Bunsubscribe@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/ __msgid/elasticsearch/f375faef-__ 2dd4-416d-8150-75afe2318f0c%__40googlegroups.com
> <https://groups.google.com/d/msgid/elasticsearch/f375faef-2dd4-416d-8150-75afe2318f0c%40googlegroups.com>.
> For more options, visit
> https://groups.google.com/__groups/opt_out
> <https://groups.google.com/groups/opt_out>.
> 
> --
> You received this message because you are subscribed to a topic in
> the Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/ __topic/elasticsearch/__ aOVumOxmsuQ/unsubscribe
> <https://groups.google.com/d/topic/elasticsearch/aOVumOxmsuQ/unsubscribe>.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscribe@__googlegroups.com
> <mailto:elasticsearch%2Bunsubscribe@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/ __msgid/elasticsearch/le4ne3%__ 24cg5%241%40ger.gmane.org
> <https://groups.google.com/d/msgid/elasticsearch/le4ne3%24cg5%241%40ger.gmane.org>.
> 
> For more options, visit https://groups.google.com/__groups/opt_out
> <https://groups.google.com/groups/opt_out>.
> 
> ```
> 
> --  
> You received this message because you are subscribed to the Google  
> Groups "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send  
> an email to  
> [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/CA%2BhvuXc%2BuQWV5ySXa0407rHCmafEef4x4OU0qdBJ5FNLzMy0yQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CA%2BhvuXc%2BuQWV5ySXa0407rHCmafEef4x4OU0qdBJ5FNLzMy0yQ%40mail.gmail.com).  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/le4t6m%24l1d%241%40ger.gmane.org](https://groups.google.com/d/msgid/elasticsearch/le4t6m%24l1d%241%40ger.gmane.org).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Tony\_Su](https://avatars.discourse-cdn.com/v4/letter/t/8dc957/32.png) [@Tony\_Su](https://discuss.elastic.co/u/Tony_Su)\
**Post date:** [February 20, 2014, 4:34pm UTC](https://discuss.elastic.co/t/migrating-to-new-cluster/15898/10 "2014-02-20T16:34:01Z")

</div>

Although it might not be clear in the documentation,

I think the strong inference is that you need to set up the new cluster  
with the same requirements for creating the snapshots in the first place.

This means...

- Shared File Repository. You need to configure every new node with the  
same access (maybe even same path) as the old cluster.
- I think you need to create your new cluster using the same exact same  
cluster and node configuration. This probably means copying your old  
elasticsearch.yml on each node to the new corresponding node.
- After doing some snapshots myself, I can see how all the data is stored  
in the backup, so although you copy data to your new cluster, I can't  
see how it would be any benefit. Assuming you intend to restore the full  
backup and not individual indices, everything will be over-written.

I'm also going to guess (maybe needs verification) there may be problems  
naming your cluster something different (needs verification but it makes  
sense).

So, I would guess that after you do your snapshots you should just shutdown  
your cluster unless it's a Production cluster. If you run both clusters  
simultaneously, then they need to be physically isolated. Then, after  
either switching or connecting your shared storage where the snapshots are  
located, execute a restore to the new cluster. If the original cluster is  
still running, then you can always make a snapshot of the changes that  
happened after the data and state has been restored to the new cluster and  
migrate the new, incremental data the same way.

IMO,  
Based on what I've observed so far...

Like doing anything else this new, IMO it would be prudent to plan for  
incremental steps in the whole process which can be back-tracked if you run  
into something unexpected.

Tony

On Thursday, February 20, 2014 4:42:11 AM UTC-8, Attila Bukor wrote:

> Hi,
> 
> I've read this one, but it's not clear to me how to do it on a different  
> machine.
> 
> If I do a backup on oldcluster with
> 
> curl -XPUT oldcluster:9200/\_snapshot/my\_backup/snapshot\_1 (assuming I've  
> already created the my\_backup repo), there will be metadata-snapshot\_1,  
> snapshot-snapshot\_1 and some other files under  
> /var/lib/elasticsearch/my\_backup or some other place which is configured.
> 
> Should I just tar the whole directory, scp it to node1 on newcluster and  
> call
> 
> curl -XPOST newcluster-node1:9200/\_snapshot/my\_backup/snapshot\_1/restore ?
> 
> I don't really want to break things and this situation is not really  
> explained in the docs.
> 
> Cheers,  
> Attila
> 
> On 02/20/2014 01:14 PM, Yann Barraud wrote:
> 
> > 
> 
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html#_restore)
> 
> > 🙂
> > 
> > Cordialement,  
> > Yann Barraud
> > 
> > 2014-02-20 12:03 GMT+01:00 Attila Bukor  
> > \<r1pp3...@w4it.eu \<javascript:\>  
> > \<[mailto:r1pp3...@w4it.eu](mailto:r1pp3...@w4it.eu) \<javascript:\>\>\>:
> > 
> > ```
> > Hi Tony, 
> > 
> > The problem is the hardware, that is why we need to move it to the 
> > 
> > ```
> 
> new
> 
> > ```
> > cluster. 
> > 
> > Yann, how can I restore a snapshot on a different machine? 
> > 
> > Cheers, 
> > Attila 
> > 
> > On 02/19/2014 07:00 PM, Tony Su wrote: 
> > 
> > I was wondering which backup/restore method you intended to use. 
> > 
> > Snapshot/restore requires 1.0. 
> > If you intend to upgrade your 0.9.x cluster to 1.0 first, it 
> > looks like 
> > possibly a really good option (I intend to start testing 
> > snapshot/restore sometime soon. I've already created snapshots 
> > and am 
> > impressed how it seems to take roughly the same amount to create 
> > 
> > ```
> 
> a
> 
> > ```
> > snapshot no matter how much data is being backed up, probably 
> > within reason) 
> > 
> > If your chosen path requires upgrading the existing cluster to 
> > 
> > ```
> 
> 1.0
> 
> > ```
> > anyway, why migrate to the new cluster? 
> > 
> > Tony 
> > 
> > On Wednesday, February 19, 2014 8:16:02 AM UTC-8, Attila Bukor 
> > wrote: 
> > 
> > Hi Yann, 
> > 
> > Thank you for your response, you saved me from a *lot* of 
> > headache. 
> > So the 
> > revised flow: 
> > 
> > - Upgrade oldcluster to 1.0.0. 
> > - Implement a logger in the application which dumps all 
> > index API calls 
> > into a file. 
> > - Backup my index on oldcluster following these docs: 
> > 
> > ```
> 
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/ __guide/en/elasticsearch/__ reference/master/modules-__snapshots.html)
> 
> > ```
> > <
> > 
> > ```
> 
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-snapshots.html)\>
> 
> > ```
> > <
> > 
> > ```
> 
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/ __guide/en/elasticsearch/__ reference/master/modules-__snapshots.html)
> 
> > ```
> > <
> > 
> > ```
> 
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-snapshots.html)\>\>
> 
> > ```
> > - Restore the backed up index on newcluster following the 
> > same docs. 
> > - Run the log from after the last _id in the snapshot. 
> > Repeat until 
> > they're the same. 
> > - Change the IP to newcluster in the application. 
> > - Repeat step 5 one more time. 
> > - Remove logging, delete logs. 
> > - Shut down oldcluster. 
> > 
> > In this case, the switch should not be noticed by anyone 
> > and the data 
> > integrity is guaranteed, right? 
> > 
> > Cheers, 
> > Attila 
> > 
> > On Wed, 19 Feb 2014, Yann Barraud wrote: 
> > 
> > > Hi, 
> > > 
> > > If I get you well, you'll face an issue. Renaming 
> > oldcluster will 
> > make it "forget" previous indexes. (Take a look at data 
> > you'll find a 
> > dir named "oldcluster"). 
> > > 
> > > The step you're missing is to backup data fro oldcluster 
> > > Then shtudown, rename 
> > > Then restore... 
> > > 
> > > Or even shutdown and restore on new cluster... 
> > > 
> > > Cheers, 
> > > Yann 
> > > 
> > > 
> > > 
> > > Le mercredi 19 février 2014 16:01:03 UTC+1, Attila Bukor 
> > a écrit : 
> > > Hey everybody, 
> > > 
> > > I have a question regarding the migration of the 
> > indices to a 
> > new cluster 
> > > seamlessly. Let me first describe the situation: 
> > > 
> > > I have a project which uses Elasticsearch since a 
> > few weeks 
> > ago. This is our 
> > > first Elasticsearch project, and as we are 
> > satisfied with it, 
> > we decided to 
> > > dedicate a cluster of 3 nodes to run Elasticsearch 
> > (lets 
> > call it 
> > > "newcluster"). It is up and running separately 
> > from our old 
> > 1-node 
> > > Elasticsearch installation ("oldcluster"). We want 
> > to migrate 
> > everything to 
> > > newcluster, and turn oldcluster off. All 4 servers 
> > are running 
> > Ubuntu Server, 
> > > Elasticsearch is installed from the official deb 
> > package, 
> > oldcluster is 
> > > 0.90.10 and newcluster 1.0.0. 
> > > 
> > > I came up with the following solution, but I'm not 
> > sure it 
> > will 
> > work: 
> > > 
> > > - Upgrade oldcluster to 1.0.0 (tested it in dev, 
> > should work) 
> > > - Backup the index (we have only one index right 
> > now) in case 
> > something goes 
> > > wrong. 
> > > - Change oldcluster's name to newcluster. 
> > > - Wait until the nodes synchronize themselves (how 
> > can I check 
> > if they're in 
> > > sync?) 
> > > - Shut down oldcluster. 
> > > 
> > > Thank you very much, 
> > > Attila 
> > > 
> > > 
> > > 
> > 
> > -- 
> > You received this message because you are subscribed to the 
> > 
> > ```
> 
> Google
> 
> > ```
> > Groups "elasticsearch" group. 
> > To unsubscribe from this group and stop receiving emails from 
> > it, send 
> > 
> > an email to 
> > elasticsearch+unsubscribe@__googlegroups.com 
> > <mailto:elasticsearch%2Bunsubscribe@googlegroups.com<javascript:>>. 
> > 
> > ```
> 
> > ```
> > To view this discussion on the web visit 
> > 
> > ```
> 
> [https://groups.google.com/d/\_\_msgid/elasticsearch/f375faef-\_\_2dd4-416d-8150-75afe2318f0c%\_\_40googlegroups.com](https://groups.google.com/d/ __msgid/elasticsearch/f375faef-__ 2dd4-416d-8150-75afe2318f0c%25__40googlegroups.com)
> 
> > ```
> > <
> > 
> > ```
> 
> [https://groups.google.com/d/msgid/elasticsearch/f375faef-2dd4-416d-8150-75afe2318f0c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f375faef-2dd4-416d-8150-75afe2318f0c%40googlegroups.com)\>.
> 
> > ```
> > For more options, visit 
> > https://groups.google.com/__groups/opt_out 
> > <https://groups.google.com/groups/opt_out>. 
> > 
> > -- 
> > You received this message because you are subscribed to a topic in 
> > the Google Groups "elasticsearch" group. 
> > To unsubscribe from this topic, visit 
> > 
> > ```
> 
> [https://groups.google.com/d/\_\_topic/elasticsearch/\_\_aOVumOxmsuQ/unsubscribe](https://groups.google.com/d/ __topic/elasticsearch/__ aOVumOxmsuQ/unsubscribe)
> 
> > ```
> > <
> > 
> > ```
> 
> [https://groups.google.com/d/topic/elasticsearch/aOVumOxmsuQ/unsubscribe](https://groups.google.com/d/topic/elasticsearch/aOVumOxmsuQ/unsubscribe)\>.
> 
> > ```
> > To unsubscribe from this group and all its topics, send an email to 
> > elasticsearch+unsubscribe@__googlegroups.com 
> > <mailto:elasticsearch%2Bunsubscribe@googlegroups.com <javascript:>>. 
> > 
> > ```
> 
> > ```
> > To view this discussion on the web visit 
> > 
> > ```
> 
> [https://groups.google.com/d/\_\_msgid/elasticsearch/le4ne3%\_\_24cg5%241%40ger.gmane.org](https://groups.google.com/d/ __msgid/elasticsearch/le4ne3%25__ 24cg5%241%40ger.gmane.org)
> 
> > ```
> > <
> > 
> > ```
> 
> [https://groups.google.com/d/msgid/elasticsearch/le4ne3%24cg5%241%40ger.gmane.org](https://groups.google.com/d/msgid/elasticsearch/le4ne3%24cg5%241%40ger.gmane.org)\>.
> 
> > ```
> > For more options, visit https://groups.google.com/__groups/opt_out 
> > <https://groups.google.com/groups/opt_out>. 
> > 
> > ```
> > 
> > --  
> > You received this message because you are subscribed to the Google  
> > Groups "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send  
> > an email to  
> > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> > To view this discussion on the web visit
> 
> [https://groups.google.com/d/msgid/elasticsearch/CA%2BhvuXc%2BuQWV5ySXa0407rHCmafEef4x4OU0qdBJ5FNLzMy0yQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CA%2BhvuXc%2BuQWV5ySXa0407rHCmafEef4x4OU0qdBJ5FNLzMy0yQ%40mail.gmail.com).
> 
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/c1e1c3e8-f2fc-43a6-bb9c-526be656eb73%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c1e1c3e8-f2fc-43a6-bb9c-526be656eb73%40googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Attila\_Bukor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/attila_bukor/32/1667_2.png) [@Attila\_Bukor](https://discuss.elastic.co/u/Attila_Bukor)\
**Post date:** [February 21, 2014, 8:41am UTC](https://discuss.elastic.co/t/migrating-to-new-cluster/15898/11 "2014-02-21T08:41:06Z")

</div>

Hey guys,

I've found these 2 threads which seem to be quite helpful in my case:

[http://elasticsearch-users.115913.n3.nabble.com/Copying-and-renaming-a-cluster-td2840624.html](http://elasticsearch-users.115913.n3.nabble.com/Copying-and-renaming-a-cluster-td2840624.html)

[http://elasticsearch-users.115913.n3.nabble.com/Changing-cluster-name-on-2-node-cluster-causes-IndexMissingException-td4035300.html](http://elasticsearch-users.115913.n3.nabble.com/Changing-cluster-name-on-2-node-cluster-causes-IndexMissingException-td4035300.html)

According to this, it should be possible to copy the whole my\_backup  
directory to newcluster and then restore.

Also, by setting "include\_global\_state" to false, I think there should be no  
problem with a different cluster name and config. Of course the config is  
the same newcluster-wide - except for the node names of course -, but it  
differs from the one node in oldcluster.

Can somebody confirm or contradict this please?

Regards,  
Attila

On 02/20/2014 05:34 PM, Tony Su wrote:

> Although it might not be clear in the documentation,  
> I think the strong inference is that you need to set up the new cluster  
> with the same requirements for creating the snapshots in the first place.  
> This means...
> 
> - Shared File Repository. You need to configure every new node with the  
> same access (maybe even same path) as the old cluster.
> - I think you need to create your new cluster using the same exact same  
> cluster and node configuration. This probably means copying your old  
> elasticsearch.yml on each node to the new corresponding node.
> - After doing some snapshots myself, I can see how all the data is  
> stored in the backup, so although you copy data to your new  
> cluster, I can't see how it would be any benefit. Assuming you intend to  
> restore the full backup and not individual indices, everything will be  
> over-written.  
> I'm also going to guess (maybe needs verification) there may be problems  
> naming your cluster something different (needs verification but it makes  
> sense).  
> So, I would guess that after you do your snapshots you should just  
> shutdown your cluster unless it's a Production cluster. If you run both  
> clusters simultaneously, then they need to be physically isolated. Then,  
> after either switching or connecting your shared storage where the  
> snapshots are located, execute a restore to the new cluster. If the  
> original cluster is still running, then you can always make a snapshot  
> of the changes that happened after the data and state has been restored  
> to the new cluster and migrate the new, incremental data the same way.  
> IMO,  
> Based on what I've observed so far...  
> Like doing anything else this new, IMO it would be prudent to plan for  
> incremental steps in the whole process which can be back-tracked if you  
> run into something unexpected.  
> Tony
> 
> On Thursday, February 20, 2014 4:42:11 AM UTC-8, Attila Bukor wrote:
> 
> ```
> Hi,
> 
> I've read this one, but it's not clear to me how to do it on a
> different
> machine.
> 
> If I do a backup on oldcluster with
> 
> curl -XPUT oldcluster:9200/_snapshot/my_backup/snapshot_1 (assuming
> I've
> already created the my_backup repo), there will be metadata-snapshot_1,
> snapshot-snapshot_1 and some other files under
> /var/lib/elasticsearch/my_backup or some other place which is
> configured.
> 
> Should I just tar the whole directory, scp it to node1 on newcluster
> and
> call
> 
> curl -XPOST
> newcluster-node1:9200/_snapshot/my_backup/snapshot_1/restore ?
> 
> I don't really want to break things and this situation is not really
> explained in the docs.
> 
> Cheers,
> Attila
> 
> On 02/20/2014 01:14 PM, Yann Barraud wrote:
> >
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html#_restore
> <http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html#_restore>
> 
> > :-)
> >
> >
> > Cordialement,
> > Yann Barraud
> >
> >
> > 2014-02-20 12:03 GMT+01:00 Attila Bukor
> > <r1pp3...@w4it.eu <javascript:>
> > <mailto:r1pp3...@w4it.eu <javascript:>>>:
> >
> > Hi Tony,
> >
> > The problem is the hardware, that is why we need to move it
> to the new
> > cluster.
> >
> > Yann, how can I restore a snapshot on a different machine?
> >
> > Cheers,
> > Attila
> >
> >
> > On 02/19/2014 07:00 PM, Tony Su wrote:
> >
> > I was wondering which backup/restore method you intended
> to use.
> >
> > Snapshot/restore requires 1.0.
> > If you intend to upgrade your 0.9.x cluster to 1.0 first, it
> > looks like
> > possibly a really good option (I intend to start testing
> > snapshot/restore sometime soon. I've already created
> snapshots
> > and am
> > impressed how it seems to take roughly the same amount to
> create a
> > snapshot no matter how much data is being backed up,
> probably
> > within reason)
> >
> > If your chosen path requires upgrading the existing
> cluster to 1.0
> > anyway, why migrate to the new cluster?
> >
> > Tony
> >
> >
> >
> > On Wednesday, February 19, 2014 8:16:02 AM UTC-8, Attila
> Bukor
> > wrote:
> >
> > Hi Yann,
> >
> > Thank you for your response, you saved me from a
> *lot* of
> > headache.
> > So the
> > revised flow:
> >
> > - Upgrade oldcluster to 1.0.0.
> > - Implement a logger in the application which dumps all
> > index API calls
> > into a file.
> > - Backup my index on oldcluster following these docs:
> >
> >
> http://www.elasticsearch.org/ __guide/en/elasticsearch/__ reference/master/modules-__snapshots.html
> <http://www.elasticsearch.org/ __guide/en/elasticsearch/__ reference/master/modules-__snapshots.html>
> 
> >
> <http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-snapshots.html
> <http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-snapshots.html>>
> 
> >
> >
> <http://www.elasticsearch.org/ __guide/en/elasticsearch/__ reference/master/modules-__snapshots.html
> <http://www.elasticsearch.org/ __guide/en/elasticsearch/__ reference/master/modules-__snapshots.html>
> 
> >
> <http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-snapshots.html
> <http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-snapshots.html>>>
> 
> >
> > - Restore the backed up index on newcluster
> following the
> > same docs.
> > - Run the log from after the last _id in the snapshot.
> > Repeat until
> > they're the same.
> > - Change the IP to newcluster in the application.
> > - Repeat step 5 one more time.
> > - Remove logging, delete logs.
> > - Shut down oldcluster.
> >
> > In this case, the switch should not be noticed by
> anyone
> > and the data
> > integrity is guaranteed, right?
> >
> > Cheers,
> > Attila
> >
> > On Wed, 19 Feb 2014, Yann Barraud wrote:
> >
> > > Hi,
> > >
> > > If I get you well, you'll face an issue. Renaming
> > oldcluster will
> > make it "forget" previous indexes. (Take a look at data
> > you'll find a
> > dir named "oldcluster").
> > >
> > > The step you're missing is to backup data fro
> oldcluster
> > > Then shtudown, rename
> > > Then restore...
> > >
> > > Or even shutdown and restore on new cluster...
> > >
> > > Cheers,
> > > Yann
> > >
> > >
> > >
> > > Le mercredi 19 février 2014 16:01:03 UTC+1,
> Attila Bukor
> > a écrit :
> > > Hey everybody,
> > >
> > > I have a question regarding the migration
> of the
> > indices to a
> > new cluster
> > > seamlessly. Let me first describe the
> situation:
> > >
> > > I have a project which uses Elasticsearch
> since a
> > few weeks
> > ago. This is our
> > > first Elasticsearch project, and as we are
> > satisfied with it,
> > we decided to
> > > dedicate a cluster of 3 nodes to run
> Elasticsearch
> > (lets
> > call it
> > > "newcluster"). It is up and running separately
> > from our old
> > 1-node
> > > Elasticsearch installation ("oldcluster").
> We want
> > to migrate
> > everything to
> > > newcluster, and turn oldcluster off. All 4
> servers
> > are running
> > Ubuntu Server,
> > > Elasticsearch is installed from the
> official deb
> > package,
> > oldcluster is
> > > 0.90.10 and newcluster 1.0.0.
> > >
> > > I came up with the following solution, but
> I'm not
> > sure it
> > will
> > work:
> > >
> > > - Upgrade oldcluster to 1.0.0 (tested it in
> dev,
> > should work)
> > > - Backup the index (we have only one index
> right
> > now) in case
> > something goes
> > > wrong.
> > > - Change oldcluster's name to newcluster.
> > > - Wait until the nodes synchronize
> themselves (how
> > can I check
> > if they're in
> > > sync?)
> > > - Shut down oldcluster.
> > >
> > > Thank you very much,
> > > Attila
> > >
> > >
> > >
> >
> >
> > --
> > You received this message because you are subscribed to
> the Google
> > Groups "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails
> from
> > it, send
> >
> > an email to
> > elasticsearch+unsubscribe@__googlegroups.com
> <http://googlegroups.com>
> >
> <mailto:elasticsearch%2Bunsubscribe@googlegroups.com
> <javascript:>>.
> > To view this discussion on the web visit
> >
> https://groups.google.com/d/ __msgid/elasticsearch/f375faef-__ 2dd4-416d-8150-75afe2318f0c%__40googlegroups.com
> <https://groups.google.com/d/ __msgid/elasticsearch/f375faef-__ 2dd4-416d-8150-75afe2318f0c%__40googlegroups.com>
> 
> >
> <https://groups.google.com/d/msgid/elasticsearch/f375faef-2dd4-416d-8150-75afe2318f0c%40googlegroups.com
> <https://groups.google.com/d/msgid/elasticsearch/f375faef-2dd4-416d-8150-75afe2318f0c%40googlegroups.com>>.
> 
> > For more options, visit
> > https://groups.google.com/__groups/opt_out
> <https://groups.google.com/__groups/opt_out>
> > <https://groups.google.com/groups/opt_out
> <https://groups.google.com/groups/opt_out>>.
> >
> >
> >
> > --
> > You received this message because you are subscribed to a
> topic in
> > the Google Groups "elasticsearch" group.
> > To unsubscribe from this topic, visit
> >
> https://groups.google.com/d/ __topic/elasticsearch/__ aOVumOxmsuQ/unsubscribe
> <https://groups.google.com/d/ __topic/elasticsearch/__ aOVumOxmsuQ/unsubscribe>
> 
> >
> <https://groups.google.com/d/topic/elasticsearch/aOVumOxmsuQ/unsubscribe
> <https://groups.google.com/d/topic/elasticsearch/aOVumOxmsuQ/unsubscribe>>.
> 
> > To unsubscribe from this group and all its topics, send an
> email to
> > elasticsearch+unsubscribe@__googlegroups.com
> <http://googlegroups.com>
> >
> <mailto:elasticsearch%2Bunsubscribe@googlegroups.com
> <javascript:>>.
> > To view this discussion on the web visit
> >
> https://groups.google.com/d/ __msgid/elasticsearch/le4ne3%__ 24cg5%241%40ger.gmane.org
> <https://groups.google.com/d/ __msgid/elasticsearch/le4ne3%__ 24cg5%241%40ger.gmane.org>
> 
> >
> <https://groups.google.com/d/msgid/elasticsearch/le4ne3%24cg5%241%40ger.gmane.org
> <https://groups.google.com/d/msgid/elasticsearch/le4ne3%24cg5%241%40ger.gmane.org>>.
> 
> >
> > For more options, visit
> https://groups.google.com/__groups/opt_out
> <https://groups.google.com/__groups/opt_out>
> > <https://groups.google.com/groups/opt_out
> <https://groups.google.com/groups/opt_out>>.
> >
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it,
> send
> > an email to
> > elasticsearc...@googlegroups.com <javascript:>.
> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/elasticsearch/CA%2BhvuXc%2BuQWV5ySXa0407rHCmafEef4x4OU0qdBJ5FNLzMy0yQ%40mail.gmail.com
> <https://groups.google.com/d/msgid/elasticsearch/CA%2BhvuXc%2BuQWV5ySXa0407rHCmafEef4x4OU0qdBJ5FNLzMy0yQ%40mail.gmail.com>.
> 
> > For more options, visit https://groups.google.com/groups/opt_out
> <https://groups.google.com/groups/opt_out>.
> 
> ```
> 
> --  
> You received this message because you are subscribed to the Google  
> Groups "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send  
> an email to  
> [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/c1e1c3e8-f2fc-43a6-bb9c-526be656eb73%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c1e1c3e8-f2fc-43a6-bb9c-526be656eb73%40googlegroups.com).  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/le73el%24pin%241%40ger.gmane.org](https://groups.google.com/d/msgid/elasticsearch/le73el%24pin%241%40ger.gmane.org).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Ivan\_Beveridge](https://avatars.discourse-cdn.com/v4/letter/i/e56c9b/32.png) [@Ivan\_Beveridge](https://discuss.elastic.co/u/Ivan_Beveridge)\
**Post date:** [February 22, 2014, 4:31pm UTC](https://discuss.elastic.co/t/migrating-to-new-cluster/15898/12 "2014-02-22T16:31:07Z")

</div>

Hi Attila,

I don't have any direct experience on this, but I'm sure I read that you  
can migrate shards from one server to another by specifying some kind of  
preference (in the elasticsearch.yml or for the index).

> **[ElasticSearch Shard Placement Control - Sematext](https://sematext.com/blog/elasticsearch-shard-placement-control/)**
>
> In this post you will learn how to control ElasticSearch shard placement.  If you are coming to our ElasticSearch talk at BerlinBuzzwords, we’ll be talking about this and more. If you’ve ever used ElasticSearch you probably know that you can set...

> **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)**
>
> Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today.

This kind of action has been previously discussed regarding migrating  
old/less-used indexes from fast servers to ones with slower disks.

With this in mind, would it not be easier to:

- expand the existing 1-node cluster (eg another 3 high-spec servers)
- ensure the existing index has at least 1 replica (or add a replica)
- configure new indexes/shards to prefer the new servers
- migrate the old shards to the new servers
- shutdown old server

Anything connecting to the ES cluster will either have to point to a new  
(loadbalanced?) IP pointing at the new nodes .... or can use the  
multicast (discovery) group address and same cluster name without  
changing anything.

Assuming I am correct (that there is a way to do this), you end up with  
no affect to the ES-cluster service.

Cheers

Ivan

On 19/02/2014 16:26, Attila Bukor wrote:

> Hi Tony,
> 
> Thank you for your response, reading this I'm pretty sure I'm going with  
> Yann's suggestion to simply backup & restore. Your first point confirms  
> that  
> this is a good idea anyway.
> 
> To answer your second point, both clusters are on the same /24 subnet, the  
> problem is that oldcluster is not designed to do that, we put a production  
> site's index on a server with twenty-odd GB free space and 4GB RAM  
> total, so  
> we _need_ to move it to the dedicated cluster.
> 
> The other two points are great suggestions and thank you very much for  
> them,  
> but I think the backup & restore way would be more suitable for my case.
> 
> Thank you very much,  
> Attila
> 
> On 02/19/2014 04:50 PM, Tony Su wrote:\> Hi,
> 
> > My testing has done a little bit of what you're describing...
> > 
> > - FWIW and I don't know why, when I re-insert the data from scratch  
> > instead of upgrading a cluster, I'm seeing substantial improvements in  
> > node general health. They seem faster, less latencies related to  
> > indexing (I assume there should be no diff searching, but ??). If a  
> > reason why I'm seeing this can't be found, then maybe what I'm seeing is  
> > specific to my setup and hardware.
> > - I don't know that there should be any reason to name your cluster  
> > differently if the networks are completely isolated, but of course if  
> > there should be a mistake, the consequences could be serious.
> > - I've been using various methods that display index activity, the  
> > easiest to use (and today work on both 0.90 and 1.0) are  
> > elasticsearch-head and elasticsearch-hq. I've also been toying around  
> > recently with simply probing each node using a curl command, which  
> > depending on how you deploy web tools likely has a lower load effect on  
> > resources (I'm currently running Apache on one node and when I use one  
> > or more of the above tools, has a significant and measurable load).
> > - If you do an upgrade, at least the one time after the upgrade you may  
> > want to force your cluster to wait until all nodes in the cluster are  
> > active before restarting to minimize shard thrashing (if a node is not  
> > online yet, the cluster may discard the indices on that node and  
> > re-allocate from whatever copy already is active. Then, when the missing  
> > node finally joins, then the shards are re-balanced). One way to do that  
> > is setting the following to be the same as the total nodes in the cluster  
> > gateway.recovery\_after\_nodes  
> > HTH,  
> > Tony
> > 
> > On Wednesday, February 19, 2014 7:01:03 AM UTC-8, Attila Bukor wrote:
> > 
> > ```
> > Hey everybody,
> > 
> > I have a question regarding the migration of the indices to a new
> > cluster
> > seamlessly. Let me first describe the situation:
> > 
> > I have a project which uses Elasticsearch since a few weeks ago.
> > This is our
> > first Elasticsearch project, and as we are satisfied with it, we
> > decided to
> > dedicate a cluster of 3 nodes to run Elasticsearch (lets call it
> > "newcluster"). It is up and running separately from our old 1-node
> > Elasticsearch installation ("oldcluster"). We want to migrate
> > everything to
> > newcluster, and turn oldcluster off. All 4 servers are running
> > Ubuntu Server,
> > Elasticsearch is installed from the official deb package,
> > 
> > ```
> 
> oldcluster is
> 
> > ```
> > 0.90.10 and newcluster 1.0.0.
> > 
> > I came up with the following solution, but I'm not sure it will work:
> > 
> > - Upgrade oldcluster to 1.0.0 (tested it in dev, should work)
> > - Backup the index (we have only one index right now) in case
> > something goes
> > wrong.
> > - Change oldcluster's name to newcluster.
> > - Wait until the nodes synchronize themselves (how can I check if
> > they're in
> > sync?)
> > - Shut down oldcluster.
> > 
> > Thank you very much,
> > Attila
> > 
> > ```

--  
Ivan Beveridge

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5308D0CB.3050609%40livejournalinc.com](https://groups.google.com/d/msgid/elasticsearch/5308D0CB.3050609%40livejournalinc.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Tony\_Su](https://avatars.discourse-cdn.com/v4/letter/t/8dc957/32.png) [@Tony\_Su](https://discuss.elastic.co/u/Tony_Su)\
**Post date:** [February 22, 2014, 5:52pm UTC](https://discuss.elastic.co/t/migrating-to-new-cluster/15898/13 "2014-02-22T17:52:34Z")

</div>

FYI -  
Within the last couple days, I attempted to do what you propose.  
You may want to experiment in a test cluster emulating your topology, too.

My steps  
Snapshot the cluster

Then I emulated a new cluster and

- Set a new cluster name in cluster.name
- Renamed the Data Directory with the new cluster name

I discovered that renaming a cluster already populated with data is no  
trivial thing. Shards were still thrashing 2hrs later when I finally gave  
up.

So, I changed courses slightly and decided to remove the existing data  
thereby emulating a new cluster.  
Since I intended to restore from the snapshots, I renamed cluster.name back  
to the original and changed the data directory to be consistent with the  
cluster name again.

Then, after re-registering the snapshot directory I did a snapshot restore  
which was 99% successful.  
I then deleted the data and restored again which this time was 100%  
successful and the cluster is still green.

Note I did not try to change the cluster name again. If you want to do so,  
you should do so as a separate step like I described so if there is an  
issue it's easy to trace and address.

Or, I would strongly recommend Ivan's suggestion. IMO creating new replicas  
is nearly trivial and doesn't put an extraordinary load on the cluster so  
you could migrate the new data to your new nodes.

Tony

On Saturday, February 22, 2014 8:31:07 AM UTC-8, Ivan Beveridge wrote:

> Hi Attila,
> 
> I don't have any direct experience on this, but I'm sure I read that you  
> can migrate shards from one server to another by specifying some kind of  
> preference (in the elasticsearch.yml or for the index).
> 
> [ElasticSearch Shard Placement Control - Sematext](http://blog.sematext.com/2012/05/29/elasticsearch-shard-placement-control/)
> 
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-cluster.html)
> 
> This kind of action has been previously discussed regarding migrating  
> old/less-used indexes from fast servers to ones with slower disks.
> 
> With this in mind, would it not be easier to:
> 
> - expand the existing 1-node cluster (eg another 3 high-spec servers)
> - ensure the existing index has at least 1 replica (or add a replica)
> - configure new indexes/shards to prefer the new servers
> - migrate the old shards to the new servers
> - shutdown old server
> 
> Anything connecting to the ES cluster will either have to point to a new  
> (loadbalanced?) IP pointing at the new nodes .... or can use the  
> multicast (discovery) group address and same cluster name without  
> changing anything.
> 
> Assuming I am correct (that there is a way to do this), you end up with  
> no affect to the ES-cluster service.
> 
> Cheers
> 
> Ivan
> 
> On 19/02/2014 16:26, Attila Bukor wrote:
> 
> > Hi Tony,
> > 
> > Thank you for your response, reading this I'm pretty sure I'm going with  
> > Yann's suggestion to simply backup & restore. Your first point confirms  
> > that  
> > this is a good idea anyway.
> > 
> > To answer your second point, both clusters are on the same /24 subnet,  
> > the  
> > problem is that oldcluster is not designed to do that, we put a  
> > production  
> > site's index on a server with twenty-odd GB free space and 4GB RAM  
> > total, so  
> > we _need_ to move it to the dedicated cluster.
> > 
> > The other two points are great suggestions and thank you very much for  
> > them,  
> > but I think the backup & restore way would be more suitable for my case.
> > 
> > Thank you very much,  
> > Attila
> > 
> > On 02/19/2014 04:50 PM, Tony Su wrote:\> Hi,
> > 
> > > My testing has done a little bit of what you're describing...
> > > 
> > > - FWIW and I don't know why, when I re-insert the data from scratch  
> > > instead of upgrading a cluster, I'm seeing substantial improvements in  
> > > node general health. They seem faster, less latencies related to  
> > > indexing (I assume there should be no diff searching, but ??). If a  
> > > reason why I'm seeing this can't be found, then maybe what I'm seeing  
> > > is  
> > > specific to my setup and hardware.
> > > - I don't know that there should be any reason to name your cluster  
> > > differently if the networks are completely isolated, but of course if  
> > > there should be a mistake, the consequences could be serious.
> > > - I've been using various methods that display index activity, the  
> > > easiest to use (and today work on both 0.90 and 1.0) are  
> > > elasticsearch-head and elasticsearch-hq. I've also been toying around  
> > > recently with simply probing each node using a curl command, which  
> > > depending on how you deploy web tools likely has a lower load effect on  
> > > resources (I'm currently running Apache on one node and when I use one  
> > > or more of the above tools, has a significant and measurable load).
> > > - If you do an upgrade, at least the one time after the upgrade you may  
> > > want to force your cluster to wait until all nodes in the cluster are  
> > > active before restarting to minimize shard thrashing (if a node is not  
> > > online yet, the cluster may discard the indices on that node and  
> > > re-allocate from whatever copy already is active. Then, when the  
> > > missing  
> > > node finally joins, then the shards are re-balanced). One way to do  
> > > that  
> > > is setting the following to be the same as the total nodes in the  
> > > cluster  
> > > gateway.recovery\_after\_nodes  
> > > HTH,  
> > > Tony
> > > 
> > > On Wednesday, February 19, 2014 7:01:03 AM UTC-8, Attila Bukor wrote:
> > > 
> > > ```
> > > Hey everybody, 
> > > 
> > > I have a question regarding the migration of the indices to a new 
> > > cluster 
> > > seamlessly. Let me first describe the situation: 
> > > 
> > > I have a project which uses Elasticsearch since a few weeks ago. 
> > > This is our 
> > > first Elasticsearch project, and as we are satisfied with it, we 
> > > decided to 
> > > dedicate a cluster of 3 nodes to run Elasticsearch (lets call it 
> > > "newcluster"). It is up and running separately from our old 1-node 
> > > Elasticsearch installation ("oldcluster"). We want to migrate 
> > > everything to 
> > > newcluster, and turn oldcluster off. All 4 servers are running 
> > > Ubuntu Server, 
> > > Elasticsearch is installed from the official deb package, 
> > > 
> > > ```
> > 
> > oldcluster is
> > 
> > > ```
> > > 0.90.10 and newcluster 1.0.0. 
> > > 
> > > I came up with the following solution, but I'm not sure it will 
> > > 
> > > ```
> 
> work:
> 
> > > ```
> > > - Upgrade oldcluster to 1.0.0 (tested it in dev, should work) 
> > > - Backup the index (we have only one index right now) in case 
> > > something goes 
> > > wrong. 
> > > - Change oldcluster's name to newcluster. 
> > > - Wait until the nodes synchronize themselves (how can I check if 
> > > they're in 
> > > sync?) 
> > > - Shut down oldcluster. 
> > > 
> > > Thank you very much, 
> > > Attila 
> > > 
> > > ```
> 
> --  
> Ivan Beveridge

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/250bc8bc-5811-420c-964b-f7d5169551a6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/250bc8bc-5811-420c-964b-f7d5169551a6%40googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:48am UTC](https://discuss.elastic.co/t/migrating-to-new-cluster/15898/14 "2017-07-06T01:48:05Z")

</div>


