Migrating to new cluster


(Attila Bukor) #1

Hey everybody,

I have a question regarding the migration of the indices to a new cluster
seamlessly. Let me first describe the situation:

I have a project which uses Elasticsearch since a few weeks ago. This is our
first Elasticsearch project, and as we are satisfied with it, we decided to
dedicate a cluster of 3 nodes to run Elasticsearch (lets call it
"newcluster"). It is up and running separately from our old 1-node
Elasticsearch installation ("oldcluster"). We want to migrate everything to
newcluster, and turn oldcluster off. All 4 servers are running Ubuntu Server,
Elasticsearch is installed from the official deb package, oldcluster is
0.90.10 and newcluster 1.0.0.

I came up with the following solution, but I'm not sure it will work:

  • Upgrade oldcluster to 1.0.0 (tested it in dev, should work)
  • Backup the index (we have only one index right now) in case something goes
    wrong.
  • Change oldcluster's name to newcluster.
  • Wait until the nodes synchronize themselves (how can I check if they're in
    sync?)
  • Shut down oldcluster.

Thank you very much,
Attila

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/alpine.LNX.2.03.1402191600190.1091%40kara.
For more options, visit https://groups.google.com/groups/opt_out.


(Yann Barraud) #2

Hi,

If I get you well, you'll face an issue. Renaming oldcluster will make it
"forget" previous indexes. (Take a look at data you'll find a dir named
"oldcluster").

The step you're missing is to backup data fro oldcluster
Then shtudown, rename
Then restore...

Or even shutdown and restore on new cluster...

Cheers,
Yann

Le mercredi 19 février 2014 16:01:03 UTC+1, Attila Bukor a écrit :

Hey everybody,

I have a question regarding the migration of the indices to a new cluster
seamlessly. Let me first describe the situation:

I have a project which uses Elasticsearch since a few weeks ago. This is
our
first Elasticsearch project, and as we are satisfied with it, we decided
to
dedicate a cluster of 3 nodes to run Elasticsearch (lets call it
"newcluster"). It is up and running separately from our old 1-node
Elasticsearch installation ("oldcluster"). We want to migrate everything
to
newcluster, and turn oldcluster off. All 4 servers are running Ubuntu
Server,
Elasticsearch is installed from the official deb package, oldcluster is
0.90.10 and newcluster 1.0.0.

I came up with the following solution, but I'm not sure it will work:

  • Upgrade oldcluster to 1.0.0 (tested it in dev, should work)
  • Backup the index (we have only one index right now) in case something
    goes
    wrong.
  • Change oldcluster's name to newcluster.
  • Wait until the nodes synchronize themselves (how can I check if they're
    in
    sync?)
  • Shut down oldcluster.

Thank you very much,
Attila

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/dc031e6c-a7d8-4773-a15d-293986331c10%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Tony Su) #3

Hi,
My testing has done a little bit of what you're describing...

  • FWIW and I don't know why, when I re-insert the data from scratch instead
    of upgrading a cluster, I'm seeing substantial improvements in node general
    health. They seem faster, less latencies related to indexing (I assume
    there should be no diff searching, but ??). If a reason why I'm seeing this
    can't be found, then maybe what I'm seeing is specific to my setup and
    hardware.

  • I don't know that there should be any reason to name your cluster
    differently if the networks are completely isolated, but of course if there
    should be a mistake, the consequences could be serious.

  • I've been using various methods that display index activity, the easiest
    to use (and today work on both 0.90 and 1.0) are elasticsearch-head and
    elasticsearch-hq. I've also been toying around recently with simply probing
    each node using a curl command, which depending on how you deploy web tools
    likely has a lower load effect on resources (I'm currently running Apache
    on one node and when I use one or more of the above tools, has a
    significant and measurable load).

  • If you do an upgrade, at least the one time after the upgrade you may
    want to force your cluster to wait until all nodes in the cluster are
    active before restarting to minimize shard thrashing (if a node is not
    online yet, the cluster may discard the indices on that node and
    re-allocate from whatever copy already is active. Then, when the missing
    node finally joins, then the shards are re-balanced). One way to do that is
    setting the following to be the same as the total nodes in the cluster

gateway.recovery_after_nodes

HTH,
Tony

On Wednesday, February 19, 2014 7:01:03 AM UTC-8, Attila Bukor wrote:

Hey everybody,

I have a question regarding the migration of the indices to a new cluster
seamlessly. Let me first describe the situation:

I have a project which uses Elasticsearch since a few weeks ago. This is
our
first Elasticsearch project, and as we are satisfied with it, we decided
to
dedicate a cluster of 3 nodes to run Elasticsearch (lets call it
"newcluster"). It is up and running separately from our old 1-node
Elasticsearch installation ("oldcluster"). We want to migrate everything
to
newcluster, and turn oldcluster off. All 4 servers are running Ubuntu
Server,
Elasticsearch is installed from the official deb package, oldcluster is
0.90.10 and newcluster 1.0.0.

I came up with the following solution, but I'm not sure it will work:

  • Upgrade oldcluster to 1.0.0 (tested it in dev, should work)
  • Backup the index (we have only one index right now) in case something
    goes
    wrong.
  • Change oldcluster's name to newcluster.
  • Wait until the nodes synchronize themselves (how can I check if they're
    in
    sync?)
  • Shut down oldcluster.

Thank you very much,
Attila

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5a1c622f-4ac5-4203-ab00-d66facb227a9%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Attila Bukor) #4

Hi Yann,

Thank you for your response, you saved me from a lot of headache. So the
revised flow:

  • Upgrade oldcluster to 1.0.0.
  • Implement a logger in the application which dumps all index API calls
    into a file.
  • Backup my index on oldcluster following these docs:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-snapshots.html

  • Restore the backed up index on newcluster following the same docs.
  • Run the log from after the last _id in the snapshot. Repeat until
    they're the same.
  • Change the IP to newcluster in the application.
  • Repeat step 5 one more time.
  • Remove logging, delete logs.
  • Shut down oldcluster.

In this case, the switch should not be noticed by anyone and the data
integrity is guaranteed, right?

Cheers,
Attila

On Wed, 19 Feb 2014, Yann Barraud wrote:

Hi,

If I get you well, you'll face an issue. Renaming oldcluster will
make it "forget" previous indexes. (Take a look at data you'll find a
dir named "oldcluster").

The step you're missing is to backup data fro oldcluster
Then shtudown, rename
Then restore...

Or even shutdown and restore on new cluster...

Cheers,
Yann

Le mercredi 19 février 2014 16:01:03 UTC+1, Attila Bukor a écrit :
Hey everybody,

  I have a question regarding the migration of the indices to a

new cluster

  seamlessly. Let me first describe the situation:

  I have a project which uses Elasticsearch since a few weeks

ago. This is our

  first Elasticsearch project, and as we are satisfied with it,

we decided to

  dedicate a cluster of 3 nodes to run Elasticsearch (lets call it
  "newcluster"). It is up and running separately from our old 1-node
  Elasticsearch installation ("oldcluster"). We want to migrate

everything to

  newcluster, and turn oldcluster off. All 4 servers are running

Ubuntu Server,

  Elasticsearch is installed from the official deb package,

oldcluster is

  0.90.10 and newcluster 1.0.0.

  I came up with the following solution, but I'm not sure it will

work:

  - Upgrade oldcluster to 1.0.0 (tested it in dev, should work)
  - Backup the index (we have only one index right now) in case

something goes

     wrong.
  - Change oldcluster's name to newcluster.
  - Wait until the nodes synchronize themselves (how can I check

if they're in

     sync?)
  - Shut down oldcluster.

  Thank you very much,
  Attila

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/le2lbk%24v8v%241%40ger.gmane.org.
For more options, visit https://groups.google.com/groups/opt_out.


(Attila Bukor) #5

Hi Tony,

Thank you for your response, reading this I'm pretty sure I'm going with
Yann's suggestion to simply backup & restore. Your first point confirms that
this is a good idea anyway.

To answer your second point, both clusters are on the same /24 subnet, the
problem is that oldcluster is not designed to do that, we put a production
site's index on a server with twenty-odd GB free space and 4GB RAM total, so
we need to move it to the dedicated cluster.

The other two points are great suggestions and thank you very much for them,
but I think the backup & restore way would be more suitable for my case.

Thank you very much,
Attila

On 02/19/2014 04:50 PM, Tony Su wrote:> Hi,

My testing has done a little bit of what you're describing...

  • FWIW and I don't know why, when I re-insert the data from scratch
    instead of upgrading a cluster, I'm seeing substantial improvements in
    node general health. They seem faster, less latencies related to
    indexing (I assume there should be no diff searching, but ??). If a
    reason why I'm seeing this can't be found, then maybe what I'm seeing is
    specific to my setup and hardware.
  • I don't know that there should be any reason to name your cluster
    differently if the networks are completely isolated, but of course if
    there should be a mistake, the consequences could be serious.
  • I've been using various methods that display index activity, the
    easiest to use (and today work on both 0.90 and 1.0) are
    elasticsearch-head and elasticsearch-hq. I've also been toying around
    recently with simply probing each node using a curl command, which
    depending on how you deploy web tools likely has a lower load effect on
    resources (I'm currently running Apache on one node and when I use one
    or more of the above tools, has a significant and measurable load).
  • If you do an upgrade, at least the one time after the upgrade you may
    want to force your cluster to wait until all nodes in the cluster are
    active before restarting to minimize shard thrashing (if a node is not
    online yet, the cluster may discard the indices on that node and
    re-allocate from whatever copy already is active. Then, when the missing
    node finally joins, then the shards are re-balanced). One way to do that
    is setting the following to be the same as the total nodes in the cluster
    gateway.recovery_after_nodes
    HTH,
    Tony

On Wednesday, February 19, 2014 7:01:03 AM UTC-8, Attila Bukor wrote:

Hey everybody,

I have a question regarding the migration of the indices to a new
cluster
seamlessly. Let me first describe the situation:

I have a project which uses Elasticsearch since a few weeks ago.
This is our
first Elasticsearch project, and as we are satisfied with it, we
decided to
dedicate a cluster of 3 nodes to run Elasticsearch (lets call it
"newcluster"). It is up and running separately from our old 1-node
Elasticsearch installation ("oldcluster"). We want to migrate
everything to
newcluster, and turn oldcluster off. All 4 servers are running
Ubuntu Server,
Elasticsearch is installed from the official deb package, 

oldcluster is

0.90.10 and newcluster 1.0.0.

I came up with the following solution, but I'm not sure it will work:

- Upgrade oldcluster to 1.0.0 (tested it in dev, should work)
- Backup the index (we have only one index right now) in case
something goes
    wrong.
- Change oldcluster's name to newcluster.
- Wait until the nodes synchronize themselves (how can I check if
they're in
    sync?)
- Shut down oldcluster.

Thank you very much,
Attila

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to

elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit

https://groups.google.com/d/msgid/elasticsearch/5a1c622f-4ac5-4203-ab00-d66facb227a9%40googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/le2luq%249t8%241%40ger.gmane.org.
For more options, visit https://groups.google.com/groups/opt_out.


(Tony Su) #6

I was wondering which backup/restore method you intended to use.

Snapshot/restore requires 1.0.
If you intend to upgrade your 0.9.x cluster to 1.0 first, it looks like
possibly a really good option (I intend to start testing snapshot/restore
sometime soon. I've already created snapshots and am impressed how it seems
to take roughly the same amount to create a snapshot no matter how much
data is being backed up, probably within reason)

If your chosen path requires upgrading the existing cluster to 1.0 anyway,
why migrate to the new cluster?

Tony

On Wednesday, February 19, 2014 8:16:02 AM UTC-8, Attila Bukor wrote:

Hi Yann,

Thank you for your response, you saved me from a lot of headache. So the
revised flow:

  • Upgrade oldcluster to 1.0.0.
  • Implement a logger in the application which dumps all index API calls
    into a file.
  • Backup my index on oldcluster following these docs:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-snapshots.html

  • Restore the backed up index on newcluster following the same docs.
  • Run the log from after the last _id in the snapshot. Repeat until
    they're the same.
  • Change the IP to newcluster in the application.
  • Repeat step 5 one more time.
  • Remove logging, delete logs.
  • Shut down oldcluster.

In this case, the switch should not be noticed by anyone and the data
integrity is guaranteed, right?

Cheers,
Attila

On Wed, 19 Feb 2014, Yann Barraud wrote:

Hi,

If I get you well, you'll face an issue. Renaming oldcluster will
make it "forget" previous indexes. (Take a look at data you'll find a
dir named "oldcluster").

The step you're missing is to backup data fro oldcluster
Then shtudown, rename
Then restore...

Or even shutdown and restore on new cluster...

Cheers,
Yann

Le mercredi 19 février 2014 16:01:03 UTC+1, Attila Bukor a écrit :
Hey everybody,

  I have a question regarding the migration of the indices to a 

new cluster

  seamlessly. Let me first describe the situation: 

  I have a project which uses Elasticsearch since a few weeks 

ago. This is our

  first Elasticsearch project, and as we are satisfied with it, 

we decided to

  dedicate a cluster of 3 nodes to run Elasticsearch (lets call it 
  "newcluster"). It is up and running separately from our old 1-node 
  Elasticsearch installation ("oldcluster"). We want to migrate 

everything to

  newcluster, and turn oldcluster off. All 4 servers are running 

Ubuntu Server,

  Elasticsearch is installed from the official deb package, 

oldcluster is

  0.90.10 and newcluster 1.0.0. 

  I came up with the following solution, but I'm not sure it will 

work:

  - Upgrade oldcluster to 1.0.0 (tested it in dev, should work) 
  - Backup the index (we have only one index right now) in case 

something goes

     wrong. 
  - Change oldcluster's name to newcluster. 
  - Wait until the nodes synchronize themselves (how can I check 

if they're in

     sync?) 
  - Shut down oldcluster. 

  Thank you very much, 
  Attila 

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f375faef-2dd4-416d-8150-75afe2318f0c%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Attila Bukor) #7

Hi Tony,

The problem is the hardware, that is why we need to move it to the new
cluster.

Yann, how can I restore a snapshot on a different machine?

Cheers,
Attila

On 02/19/2014 07:00 PM, Tony Su wrote:

I was wondering which backup/restore method you intended to use.

Snapshot/restore requires 1.0.
If you intend to upgrade your 0.9.x cluster to 1.0 first, it looks like
possibly a really good option (I intend to start testing
snapshot/restore sometime soon. I've already created snapshots and am
impressed how it seems to take roughly the same amount to create a
snapshot no matter how much data is being backed up, probably within reason)

If your chosen path requires upgrading the existing cluster to 1.0
anyway, why migrate to the new cluster?

Tony

On Wednesday, February 19, 2014 8:16:02 AM UTC-8, Attila Bukor wrote:

Hi Yann,

Thank you for your response, you saved me from a *lot* of headache.
So the
revised flow:

- Upgrade oldcluster to 1.0.0.
- Implement a logger in the application which dumps all index API calls
    into a file.
- Backup my index on oldcluster following these docs:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-snapshots.html
<http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-snapshots.html>

- Restore the backed up index on newcluster following the same docs.
- Run the log from after the last _id in the snapshot. Repeat until
    they're the same.
- Change the IP to newcluster in the application.
- Repeat step 5 one more time.
- Remove logging, delete logs.
- Shut down oldcluster.

In this case, the switch should not be noticed by anyone and the data
integrity is guaranteed, right?

Cheers,
Attila

On Wed, 19 Feb 2014, Yann Barraud wrote:

 > Hi,
 >
 > If I get you well, you'll face an issue. Renaming oldcluster will
make it "forget" previous indexes. (Take a look at data you'll find a
dir named "oldcluster").
 >
 > The step you're missing is to backup data fro oldcluster
 > Then shtudown, rename
 > Then restore...
 >
 > Or even shutdown and restore on new cluster...
 >
 > Cheers,
 > Yann
 >
 >
 >
 > Le mercredi 19 février 2014 16:01:03 UTC+1, Attila Bukor a écrit :
 >       Hey everybody,
 >
 >       I have a question regarding the migration of the indices to a
new cluster
 >       seamlessly. Let me first describe the situation:
 >
 >       I have a project which uses Elasticsearch since a few weeks
ago. This is our
 >       first Elasticsearch project, and as we are satisfied with it,
we decided to
 >       dedicate a cluster of 3 nodes to run Elasticsearch (lets
call it
 >       "newcluster"). It is up and running separately from our old
1-node
 >       Elasticsearch installation ("oldcluster"). We want to migrate
everything to
 >       newcluster, and turn oldcluster off. All 4 servers are running
Ubuntu Server,
 >       Elasticsearch is installed from the official deb package,
oldcluster is
 >       0.90.10 and newcluster 1.0.0.
 >
 >       I came up with the following solution, but I'm not sure it
will
work:
 >
 >       - Upgrade oldcluster to 1.0.0 (tested it in dev, should work)
 >       - Backup the index (we have only one index right now) in case
something goes
 >          wrong.
 >       - Change oldcluster's name to newcluster.
 >       - Wait until the nodes synchronize themselves (how can I check
if they're in
 >          sync?)
 >       - Shut down oldcluster.
 >
 >       Thank you very much,
 >       Attila
 >
 >
 >

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/f375faef-2dd4-416d-8150-75afe2318f0c%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/le4ne3%24cg5%241%40ger.gmane.org.
For more options, visit https://groups.google.com/groups/opt_out.


(Yann Barraud) #8

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html#_restore:-)

Cordialement,
Yann Barraud

2014-02-20 12:03 GMT+01:00 Attila Bukor r1pp3rj4ck@w4it.eu:

Hi Tony,

The problem is the hardware, that is why we need to move it to the new
cluster.

Yann, how can I restore a snapshot on a different machine?

Cheers,
Attila

On 02/19/2014 07:00 PM, Tony Su wrote:

I was wondering which backup/restore method you intended to use.

Snapshot/restore requires 1.0.
If you intend to upgrade your 0.9.x cluster to 1.0 first, it looks like
possibly a really good option (I intend to start testing
snapshot/restore sometime soon. I've already created snapshots and am
impressed how it seems to take roughly the same amount to create a
snapshot no matter how much data is being backed up, probably within
reason)

If your chosen path requires upgrading the existing cluster to 1.0
anyway, why migrate to the new cluster?

Tony

On Wednesday, February 19, 2014 8:16:02 AM UTC-8, Attila Bukor wrote:

Hi Yann,

Thank you for your response, you saved me from a *lot* of headache.
So the
revised flow:

- Upgrade oldcluster to 1.0.0.
- Implement a logger in the application which dumps all index API

calls
into a file.
- Backup my index on oldcluster following these docs:

http://www.elasticsearch.org/guide/en/elasticsearch/

reference/master/modules-snapshots.html
<http://www.elasticsearch.org/guide/en/elasticsearch/
reference/master/modules-snapshots.html>

- Restore the backed up index on newcluster following the same docs.
- Run the log from after the last _id in the snapshot. Repeat until
    they're the same.
- Change the IP to newcluster in the application.
- Repeat step 5 one more time.
- Remove logging, delete logs.
- Shut down oldcluster.

In this case, the switch should not be noticed by anyone and the data
integrity is guaranteed, right?

Cheers,
Attila

On Wed, 19 Feb 2014, Yann Barraud wrote:

 > Hi,
 >
 > If I get you well, you'll face an issue. Renaming oldcluster will
make it "forget" previous indexes. (Take a look at data you'll find a
dir named "oldcluster").
 >
 > The step you're missing is to backup data fro oldcluster
 > Then shtudown, rename
 > Then restore...
 >
 > Or even shutdown and restore on new cluster...
 >
 > Cheers,
 > Yann
 >
 >
 >
 > Le mercredi 19 février 2014 16:01:03 UTC+1, Attila Bukor a écrit :
 >       Hey everybody,
 >
 >       I have a question regarding the migration of the indices to a
new cluster
 >       seamlessly. Let me first describe the situation:
 >
 >       I have a project which uses Elasticsearch since a few weeks
ago. This is our
 >       first Elasticsearch project, and as we are satisfied with it,
we decided to
 >       dedicate a cluster of 3 nodes to run Elasticsearch (lets
call it
 >       "newcluster"). It is up and running separately from our old
1-node
 >       Elasticsearch installation ("oldcluster"). We want to migrate
everything to
 >       newcluster, and turn oldcluster off. All 4 servers are

running
Ubuntu Server,
> Elasticsearch is installed from the official deb package,
oldcluster is
> 0.90.10 and newcluster 1.0.0.
>
> I came up with the following solution, but I'm not sure it
will
work:
>
> - Upgrade oldcluster to 1.0.0 (tested it in dev, should work)
> - Backup the index (we have only one index right now) in case
something goes
> wrong.
> - Change oldcluster's name to newcluster.
> - Wait until the nodes synchronize themselves (how can I
check
if they're in
> sync?)
> - Shut down oldcluster.
>
> Thank you very much,
> Attila
>
>
>

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send

an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/f375faef-
2dd4-416d-8150-75afe2318f0c%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/
topic/elasticsearch/aOVumOxmsuQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/le4ne3%24cg5%241%40ger.gmane.org.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CA%2BhvuXc%2BuQWV5ySXa0407rHCmafEef4x4OU0qdBJ5FNLzMy0yQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Attila Bukor) #9

Hi,

I've read this one, but it's not clear to me how to do it on a different
machine.

If I do a backup on oldcluster with

curl -XPUT oldcluster:9200/_snapshot/my_backup/snapshot_1 (assuming I've
already created the my_backup repo), there will be metadata-snapshot_1,
snapshot-snapshot_1 and some other files under
/var/lib/elasticsearch/my_backup or some other place which is configured.

Should I just tar the whole directory, scp it to node1 on newcluster and
call

curl -XPOST newcluster-node1:9200/_snapshot/my_backup/snapshot_1/restore ?

I don't really want to break things and this situation is not really
explained in the docs.

Cheers,
Attila

On 02/20/2014 01:14 PM, Yann Barraud wrote:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html#_restore
:slight_smile:

Cordialement,
Yann Barraud

2014-02-20 12:03 GMT+01:00 Attila Bukor
<r1pp3rj4ck@w4it.eu
mailto:r1pp3rj4ck@w4it.eu>:

Hi Tony,

The problem is the hardware, that is why we need to move it to the new
cluster.

Yann, how can I restore a snapshot on a different machine?

Cheers,
Attila


On 02/19/2014 07:00 PM, Tony Su wrote:

    I was wondering which backup/restore method you intended to use.

    Snapshot/restore requires 1.0.
    If you intend to upgrade your 0.9.x cluster to 1.0 first, it
    looks like
    possibly a really good option (I intend to start testing
    snapshot/restore sometime soon. I've already created snapshots
    and am
    impressed how it seems to take roughly the same amount to create a
    snapshot no matter how much data is being backed up, probably
    within reason)

    If your chosen path requires upgrading the existing cluster to 1.0
    anyway, why migrate to the new cluster?

    Tony



    On Wednesday, February 19, 2014 8:16:02 AM UTC-8, Attila Bukor
    wrote:

         Hi Yann,

         Thank you for your response, you saved me from a *lot* of
    headache.
         So the
         revised flow:

         - Upgrade oldcluster to 1.0.0.
         - Implement a logger in the application which dumps all
    index API calls
             into a file.
         - Backup my index on oldcluster following these docs:

    http://www.elasticsearch.org/__guide/en/elasticsearch/__reference/master/modules-__snapshots.html
    <http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-snapshots.html>

    <http://www.elasticsearch.org/__guide/en/elasticsearch/__reference/master/modules-__snapshots.html
    <http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-snapshots.html>>

         - Restore the backed up index on newcluster following the
    same docs.
         - Run the log from after the last _id in the snapshot.
    Repeat until
             they're the same.
         - Change the IP to newcluster in the application.
         - Repeat step 5 one more time.
         - Remove logging, delete logs.
         - Shut down oldcluster.

         In this case, the switch should not be noticed by anyone
    and the data
         integrity is guaranteed, right?

         Cheers,
         Attila

         On Wed, 19 Feb 2014, Yann Barraud wrote:

          > Hi,
          >
          > If I get you well, you'll face an issue. Renaming
    oldcluster will
         make it "forget" previous indexes. (Take a look at data
    you'll find a
         dir named "oldcluster").
          >
          > The step you're missing is to backup data fro oldcluster
          > Then shtudown, rename
          > Then restore...
          >
          > Or even shutdown and restore on new cluster...
          >
          > Cheers,
          > Yann
          >
          >
          >
          > Le mercredi 19 février 2014 16:01:03 UTC+1, Attila Bukor
    a écrit :
          >       Hey everybody,
          >
          >       I have a question regarding the migration of the
    indices to a
         new cluster
          >       seamlessly. Let me first describe the situation:
          >
          >       I have a project which uses Elasticsearch since a
    few weeks
         ago. This is our
          >       first Elasticsearch project, and as we are
    satisfied with it,
         we decided to
          >       dedicate a cluster of 3 nodes to run Elasticsearch
    (lets
         call it
          >       "newcluster"). It is up and running separately
    from our old
         1-node
          >       Elasticsearch installation ("oldcluster"). We want
    to migrate
         everything to
          >       newcluster, and turn oldcluster off. All 4 servers
    are running
         Ubuntu Server,
          >       Elasticsearch is installed from the official deb
    package,
         oldcluster is
          >       0.90.10 and newcluster 1.0.0.
          >
          >       I came up with the following solution, but I'm not
    sure it
         will
         work:
          >
          >       - Upgrade oldcluster to 1.0.0 (tested it in dev,
    should work)
          >       - Backup the index (we have only one index right
    now) in case
         something goes
          >          wrong.
          >       - Change oldcluster's name to newcluster.
          >       - Wait until the nodes synchronize themselves (how
    can I check
         if they're in
          >          sync?)
          >       - Shut down oldcluster.
          >
          >       Thank you very much,
          >       Attila
          >
          >
          >


    --
    You received this message because you are subscribed to the Google
    Groups "elasticsearch" group.
    To unsubscribe from this group and stop receiving emails from
    it, send

    an email to
    elasticsearch+unsubscribe@__googlegroups.com
    <mailto:elasticsearch%2Bunsubscribe@googlegroups.com>.
    To view this discussion on the web visit
    https://groups.google.com/d/__msgid/elasticsearch/f375faef-__2dd4-416d-8150-75afe2318f0c%__40googlegroups.com
    <https://groups.google.com/d/msgid/elasticsearch/f375faef-2dd4-416d-8150-75afe2318f0c%40googlegroups.com>.
    For more options, visit
    https://groups.google.com/__groups/opt_out
    <https://groups.google.com/groups/opt_out>.



--
You received this message because you are subscribed to a topic in
the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/__topic/elasticsearch/__aOVumOxmsuQ/unsubscribe
<https://groups.google.com/d/topic/elasticsearch/aOVumOxmsuQ/unsubscribe>.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@__googlegroups.com
<mailto:elasticsearch%2Bunsubscribe@googlegroups.com>.
To view this discussion on the web visit
https://groups.google.com/d/__msgid/elasticsearch/le4ne3%__24cg5%241%40ger.gmane.org
<https://groups.google.com/d/msgid/elasticsearch/le4ne3%24cg5%241%40ger.gmane.org>.

For more options, visit https://groups.google.com/__groups/opt_out
<https://groups.google.com/groups/opt_out>.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CA%2BhvuXc%2BuQWV5ySXa0407rHCmafEef4x4OU0qdBJ5FNLzMy0yQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/le4t6m%24l1d%241%40ger.gmane.org.
For more options, visit https://groups.google.com/groups/opt_out.


(Tony Su) #10

Although it might not be clear in the documentation,

I think the strong inference is that you need to set up the new cluster
with the same requirements for creating the snapshots in the first place.

This means...

  • Shared File Repository. You need to configure every new node with the
    same access (maybe even same path) as the old cluster.
  • I think you need to create your new cluster using the same exact same
    cluster and node configuration. This probably means copying your old
    elasticsearch.yml on each node to the new corresponding node.
  • After doing some snapshots myself, I can see how all the data is stored
    in the backup, so although you copy data to your new cluster, I can't
    see how it would be any benefit. Assuming you intend to restore the full
    backup and not individual indices, everything will be over-written.

I'm also going to guess (maybe needs verification) there may be problems
naming your cluster something different (needs verification but it makes
sense).

So, I would guess that after you do your snapshots you should just shutdown
your cluster unless it's a Production cluster. If you run both clusters
simultaneously, then they need to be physically isolated. Then, after
either switching or connecting your shared storage where the snapshots are
located, execute a restore to the new cluster. If the original cluster is
still running, then you can always make a snapshot of the changes that
happened after the data and state has been restored to the new cluster and
migrate the new, incremental data the same way.

IMO,
Based on what I've observed so far...

Like doing anything else this new, IMO it would be prudent to plan for
incremental steps in the whole process which can be back-tracked if you run
into something unexpected.

Tony

On Thursday, February 20, 2014 4:42:11 AM UTC-8, Attila Bukor wrote:

Hi,

I've read this one, but it's not clear to me how to do it on a different
machine.

If I do a backup on oldcluster with

curl -XPUT oldcluster:9200/_snapshot/my_backup/snapshot_1 (assuming I've
already created the my_backup repo), there will be metadata-snapshot_1,
snapshot-snapshot_1 and some other files under
/var/lib/elasticsearch/my_backup or some other place which is configured.

Should I just tar the whole directory, scp it to node1 on newcluster and
call

curl -XPOST newcluster-node1:9200/_snapshot/my_backup/snapshot_1/restore ?

I don't really want to break things and this situation is not really
explained in the docs.

Cheers,
Attila

On 02/20/2014 01:14 PM, Yann Barraud wrote:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html#_restore

:slight_smile:

Cordialement,
Yann Barraud

2014-02-20 12:03 GMT+01:00 Attila Bukor
<r1pp3...@w4it.eu <javascript:>
<mailto:r1pp3...@w4it.eu <javascript:>>>:

Hi Tony, 

The problem is the hardware, that is why we need to move it to the 

new

cluster. 

Yann, how can I restore a snapshot on a different machine? 

Cheers, 
Attila 


On 02/19/2014 07:00 PM, Tony Su wrote: 

    I was wondering which backup/restore method you intended to use. 

    Snapshot/restore requires 1.0. 
    If you intend to upgrade your 0.9.x cluster to 1.0 first, it 
    looks like 
    possibly a really good option (I intend to start testing 
    snapshot/restore sometime soon. I've already created snapshots 
    and am 
    impressed how it seems to take roughly the same amount to create 

a

    snapshot no matter how much data is being backed up, probably 
    within reason) 

    If your chosen path requires upgrading the existing cluster to 

1.0

    anyway, why migrate to the new cluster? 

    Tony 



    On Wednesday, February 19, 2014 8:16:02 AM UTC-8, Attila Bukor 
    wrote: 

         Hi Yann, 

         Thank you for your response, you saved me from a *lot* of 
    headache. 
         So the 
         revised flow: 

         - Upgrade oldcluster to 1.0.0. 
         - Implement a logger in the application which dumps all 
    index API calls 
             into a file. 
         - Backup my index on oldcluster following these docs: 

http://www.elasticsearch.org/__guide/en/elasticsearch/__reference/master/modules-__snapshots.html

    <

http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-snapshots.html>

    <

http://www.elasticsearch.org/__guide/en/elasticsearch/__reference/master/modules-__snapshots.html

    <

http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-snapshots.html>>

         - Restore the backed up index on newcluster following the 
    same docs. 
         - Run the log from after the last _id in the snapshot. 
    Repeat until 
             they're the same. 
         - Change the IP to newcluster in the application. 
         - Repeat step 5 one more time. 
         - Remove logging, delete logs. 
         - Shut down oldcluster. 

         In this case, the switch should not be noticed by anyone 
    and the data 
         integrity is guaranteed, right? 

         Cheers, 
         Attila 

         On Wed, 19 Feb 2014, Yann Barraud wrote: 

          > Hi, 
          > 
          > If I get you well, you'll face an issue. Renaming 
    oldcluster will 
         make it "forget" previous indexes. (Take a look at data 
    you'll find a 
         dir named "oldcluster"). 
          > 
          > The step you're missing is to backup data fro oldcluster 
          > Then shtudown, rename 
          > Then restore... 
          > 
          > Or even shutdown and restore on new cluster... 
          > 
          > Cheers, 
          > Yann 
          > 
          > 
          > 
          > Le mercredi 19 février 2014 16:01:03 UTC+1, Attila Bukor 
    a écrit : 
          >       Hey everybody, 
          > 
          >       I have a question regarding the migration of the 
    indices to a 
         new cluster 
          >       seamlessly. Let me first describe the situation: 
          > 
          >       I have a project which uses Elasticsearch since a 
    few weeks 
         ago. This is our 
          >       first Elasticsearch project, and as we are 
    satisfied with it, 
         we decided to 
          >       dedicate a cluster of 3 nodes to run Elasticsearch 
    (lets 
         call it 
          >       "newcluster"). It is up and running separately 
    from our old 
         1-node 
          >       Elasticsearch installation ("oldcluster"). We want 
    to migrate 
         everything to 
          >       newcluster, and turn oldcluster off. All 4 servers 
    are running 
         Ubuntu Server, 
          >       Elasticsearch is installed from the official deb 
    package, 
         oldcluster is 
          >       0.90.10 and newcluster 1.0.0. 
          > 
          >       I came up with the following solution, but I'm not 
    sure it 
         will 
         work: 
          > 
          >       - Upgrade oldcluster to 1.0.0 (tested it in dev, 
    should work) 
          >       - Backup the index (we have only one index right 
    now) in case 
         something goes 
          >          wrong. 
          >       - Change oldcluster's name to newcluster. 
          >       - Wait until the nodes synchronize themselves (how 
    can I check 
         if they're in 
          >          sync?) 
          >       - Shut down oldcluster. 
          > 
          >       Thank you very much, 
          >       Attila 
          > 
          > 
          > 


    -- 
    You received this message because you are subscribed to the 

Google

    Groups "elasticsearch" group. 
    To unsubscribe from this group and stop receiving emails from 
    it, send 

    an email to 
    elasticsearch+unsubscribe@__googlegroups.com 
    <mailto:elasticsearch%2Bunsubscribe@googlegroups.com<javascript:>>. 
    To view this discussion on the web visit 

https://groups.google.com/d/__msgid/elasticsearch/f375faef-__2dd4-416d-8150-75afe2318f0c%__40googlegroups.com

    <

https://groups.google.com/d/msgid/elasticsearch/f375faef-2dd4-416d-8150-75afe2318f0c%40googlegroups.com>.

    For more options, visit 
    https://groups.google.com/__groups/opt_out 
    <https://groups.google.com/groups/opt_out>. 



-- 
You received this message because you are subscribed to a topic in 
the Google Groups "elasticsearch" group. 
To unsubscribe from this topic, visit 

https://groups.google.com/d/__topic/elasticsearch/__aOVumOxmsuQ/unsubscribe

<

https://groups.google.com/d/topic/elasticsearch/aOVumOxmsuQ/unsubscribe>.

To unsubscribe from this group and all its topics, send an email to 
elasticsearch+unsubscribe@__googlegroups.com 
<mailto:elasticsearch%2Bunsubscribe@googlegroups.com <javascript:>>. 
To view this discussion on the web visit 

https://groups.google.com/d/__msgid/elasticsearch/le4ne3%__24cg5%241%40ger.gmane.org

<

https://groups.google.com/d/msgid/elasticsearch/le4ne3%24cg5%241%40ger.gmane.org>.

For more options, visit https://groups.google.com/__groups/opt_out 
<https://groups.google.com/groups/opt_out>. 

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to
elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit

https://groups.google.com/d/msgid/elasticsearch/CA%2BhvuXc%2BuQWV5ySXa0407rHCmafEef4x4OU0qdBJ5FNLzMy0yQ%40mail.gmail.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c1e1c3e8-f2fc-43a6-bb9c-526be656eb73%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Attila Bukor) #11

Hey guys,

I've found these 2 threads which seem to be quite helpful in my case:

http://elasticsearch-users.115913.n3.nabble.com/Copying-and-renaming-a-cluster-td2840624.html

http://elasticsearch-users.115913.n3.nabble.com/Changing-cluster-name-on-2-node-cluster-causes-IndexMissingException-td4035300.html

According to this, it should be possible to copy the whole my_backup
directory to newcluster and then restore.

Also, by setting "include_global_state" to false, I think there should be no
problem with a different cluster name and config. Of course the config is
the same newcluster-wide - except for the node names of course -, but it
differs from the one node in oldcluster.

Can somebody confirm or contradict this please?

Regards,
Attila

On 02/20/2014 05:34 PM, Tony Su wrote:

Although it might not be clear in the documentation,
I think the strong inference is that you need to set up the new cluster
with the same requirements for creating the snapshots in the first place.
This means...

  • Shared File Repository. You need to configure every new node with the
    same access (maybe even same path) as the old cluster.
  • I think you need to create your new cluster using the same exact same
    cluster and node configuration. This probably means copying your old
    elasticsearch.yml on each node to the new corresponding node.
  • After doing some snapshots myself, I can see how all the data is
    stored in the backup, so although you copy data to your new
    cluster, I can't see how it would be any benefit. Assuming you intend to
    restore the full backup and not individual indices, everything will be
    over-written.
    I'm also going to guess (maybe needs verification) there may be problems
    naming your cluster something different (needs verification but it makes
    sense).
    So, I would guess that after you do your snapshots you should just
    shutdown your cluster unless it's a Production cluster. If you run both
    clusters simultaneously, then they need to be physically isolated. Then,
    after either switching or connecting your shared storage where the
    snapshots are located, execute a restore to the new cluster. If the
    original cluster is still running, then you can always make a snapshot
    of the changes that happened after the data and state has been restored
    to the new cluster and migrate the new, incremental data the same way.
    IMO,
    Based on what I've observed so far...
    Like doing anything else this new, IMO it would be prudent to plan for
    incremental steps in the whole process which can be back-tracked if you
    run into something unexpected.
    Tony

On Thursday, February 20, 2014 4:42:11 AM UTC-8, Attila Bukor wrote:

Hi,

I've read this one, but it's not clear to me how to do it on a
different
machine.

If I do a backup on oldcluster with

curl -XPUT oldcluster:9200/_snapshot/my_backup/snapshot_1 (assuming
I've
already created the my_backup repo), there will be metadata-snapshot_1,
snapshot-snapshot_1 and some other files under
/var/lib/elasticsearch/my_backup or some other place which is
configured.

Should I just tar the whole directory, scp it to node1 on newcluster
and
call

curl -XPOST
newcluster-node1:9200/_snapshot/my_backup/snapshot_1/restore ?

I don't really want to break things and this situation is not really
explained in the docs.

Cheers,
Attila

On 02/20/2014 01:14 PM, Yann Barraud wrote:
 >
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html#_restore
<http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html#_restore>

 > :-)
 >
 >
 > Cordialement,
 > Yann Barraud
 >
 >
 > 2014-02-20 12:03 GMT+01:00 Attila Bukor
 > <r1pp3...@w4it.eu <javascript:>
 > <mailto:r1pp3...@w4it.eu <javascript:>>>:
 >
 >     Hi Tony,
 >
 >     The problem is the hardware, that is why we need to move it
to the new
 >     cluster.
 >
 >     Yann, how can I restore a snapshot on a different machine?
 >
 >     Cheers,
 >     Attila
 >
 >
 >     On 02/19/2014 07:00 PM, Tony Su wrote:
 >
 >         I was wondering which backup/restore method you intended
to use.
 >
 >         Snapshot/restore requires 1.0.
 >         If you intend to upgrade your 0.9.x cluster to 1.0 first, it
 >         looks like
 >         possibly a really good option (I intend to start testing
 >         snapshot/restore sometime soon. I've already created
snapshots
 >         and am
 >         impressed how it seems to take roughly the same amount to
create a
 >         snapshot no matter how much data is being backed up,
probably
 >         within reason)
 >
 >         If your chosen path requires upgrading the existing
cluster to 1.0
 >         anyway, why migrate to the new cluster?
 >
 >         Tony
 >
 >
 >
 >         On Wednesday, February 19, 2014 8:16:02 AM UTC-8, Attila
Bukor
 >         wrote:
 >
 >              Hi Yann,
 >
 >              Thank you for your response, you saved me from a
*lot* of
 >         headache.
 >              So the
 >              revised flow:
 >
 >              - Upgrade oldcluster to 1.0.0.
 >              - Implement a logger in the application which dumps all
 >         index API calls
 >                  into a file.
 >              - Backup my index on oldcluster following these docs:
 >
 >
http://www.elasticsearch.org/__guide/en/elasticsearch/__reference/master/modules-__snapshots.html
<http://www.elasticsearch.org/__guide/en/elasticsearch/__reference/master/modules-__snapshots.html>

 >
<http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-snapshots.html
<http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-snapshots.html>>

 >
 >
<http://www.elasticsearch.org/__guide/en/elasticsearch/__reference/master/modules-__snapshots.html
<http://www.elasticsearch.org/__guide/en/elasticsearch/__reference/master/modules-__snapshots.html>

 >
<http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-snapshots.html
<http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-snapshots.html>>>

 >
 >              - Restore the backed up index on newcluster
following the
 >         same docs.
 >              - Run the log from after the last _id in the snapshot.
 >         Repeat until
 >                  they're the same.
 >              - Change the IP to newcluster in the application.
 >              - Repeat step 5 one more time.
 >              - Remove logging, delete logs.
 >              - Shut down oldcluster.
 >
 >              In this case, the switch should not be noticed by
anyone
 >         and the data
 >              integrity is guaranteed, right?
 >
 >              Cheers,
 >              Attila
 >
 >              On Wed, 19 Feb 2014, Yann Barraud wrote:
 >
 >               > Hi,
 >               >
 >               > If I get you well, you'll face an issue. Renaming
 >         oldcluster will
 >              make it "forget" previous indexes. (Take a look at data
 >         you'll find a
 >              dir named "oldcluster").
 >               >
 >               > The step you're missing is to backup data fro
oldcluster
 >               > Then shtudown, rename
 >               > Then restore...
 >               >
 >               > Or even shutdown and restore on new cluster...
 >               >
 >               > Cheers,
 >               > Yann
 >               >
 >               >
 >               >
 >               > Le mercredi 19 février 2014 16:01:03 UTC+1,
Attila Bukor
 >         a écrit :
 >               >       Hey everybody,
 >               >
 >               >       I have a question regarding the migration
of the
 >         indices to a
 >              new cluster
 >               >       seamlessly. Let me first describe the
situation:
 >               >
 >               >       I have a project which uses Elasticsearch
since a
 >         few weeks
 >              ago. This is our
 >               >       first Elasticsearch project, and as we are
 >         satisfied with it,
 >              we decided to
 >               >       dedicate a cluster of 3 nodes to run
Elasticsearch
 >         (lets
 >              call it
 >               >       "newcluster"). It is up and running separately
 >         from our old
 >              1-node
 >               >       Elasticsearch installation ("oldcluster").
We want
 >         to migrate
 >              everything to
 >               >       newcluster, and turn oldcluster off. All 4
servers
 >         are running
 >              Ubuntu Server,
 >               >       Elasticsearch is installed from the
official deb
 >         package,
 >              oldcluster is
 >               >       0.90.10 and newcluster 1.0.0.
 >               >
 >               >       I came up with the following solution, but
I'm not
 >         sure it
 >              will
 >              work:
 >               >
 >               >       - Upgrade oldcluster to 1.0.0 (tested it in
dev,
 >         should work)
 >               >       - Backup the index (we have only one index
right
 >         now) in case
 >              something goes
 >               >          wrong.
 >               >       - Change oldcluster's name to newcluster.
 >               >       - Wait until the nodes synchronize
themselves (how
 >         can I check
 >              if they're in
 >               >          sync?)
 >               >       - Shut down oldcluster.
 >               >
 >               >       Thank you very much,
 >               >       Attila
 >               >
 >               >
 >               >
 >
 >
 >         --
 >         You received this message because you are subscribed to
the Google
 >         Groups "elasticsearch" group.
 >         To unsubscribe from this group and stop receiving emails
from
 >         it, send
 >
 >         an email to
 >         elasticsearch+unsubscribe@__googlegroups.com
<http://googlegroups.com>
 >
<mailto:elasticsearch%2Bunsubscribe@googlegroups.com
<javascript:>>.
 >         To view this discussion on the web visit
 >
https://groups.google.com/d/__msgid/elasticsearch/f375faef-__2dd4-416d-8150-75afe2318f0c%__40googlegroups.com
<https://groups.google.com/d/__msgid/elasticsearch/f375faef-__2dd4-416d-8150-75afe2318f0c%__40googlegroups.com>

 >
<https://groups.google.com/d/msgid/elasticsearch/f375faef-2dd4-416d-8150-75afe2318f0c%40googlegroups.com
<https://groups.google.com/d/msgid/elasticsearch/f375faef-2dd4-416d-8150-75afe2318f0c%40googlegroups.com>>.

 >         For more options, visit
 > https://groups.google.com/__groups/opt_out
<https://groups.google.com/__groups/opt_out>
 >         <https://groups.google.com/groups/opt_out
<https://groups.google.com/groups/opt_out>>.
 >
 >
 >
 >     --
 >     You received this message because you are subscribed to a
topic in
 >     the Google Groups "elasticsearch" group.
 >     To unsubscribe from this topic, visit
 >
https://groups.google.com/d/__topic/elasticsearch/__aOVumOxmsuQ/unsubscribe
<https://groups.google.com/d/__topic/elasticsearch/__aOVumOxmsuQ/unsubscribe>

 >
<https://groups.google.com/d/topic/elasticsearch/aOVumOxmsuQ/unsubscribe
<https://groups.google.com/d/topic/elasticsearch/aOVumOxmsuQ/unsubscribe>>.

 >     To unsubscribe from this group and all its topics, send an
email to
 >     elasticsearch+unsubscribe@__googlegroups.com
<http://googlegroups.com>
 >
<mailto:elasticsearch%2Bunsubscribe@googlegroups.com
<javascript:>>.
 >     To view this discussion on the web visit
 >
https://groups.google.com/d/__msgid/elasticsearch/le4ne3%__24cg5%241%40ger.gmane.org
<https://groups.google.com/d/__msgid/elasticsearch/le4ne3%__24cg5%241%40ger.gmane.org>

 >
<https://groups.google.com/d/msgid/elasticsearch/le4ne3%24cg5%241%40ger.gmane.org
<https://groups.google.com/d/msgid/elasticsearch/le4ne3%24cg5%241%40ger.gmane.org>>.

 >
 >     For more options, visit
https://groups.google.com/__groups/opt_out
<https://groups.google.com/__groups/opt_out>
 >     <https://groups.google.com/groups/opt_out
<https://groups.google.com/groups/opt_out>>.
 >
 >
 > --
 > You received this message because you are subscribed to the Google
 > Groups "elasticsearch" group.
 > To unsubscribe from this group and stop receiving emails from it,
send
 > an email to
 > elasticsearc...@googlegroups.com <javascript:>.
 > To view this discussion on the web visit
 >
https://groups.google.com/d/msgid/elasticsearch/CA%2BhvuXc%2BuQWV5ySXa0407rHCmafEef4x4OU0qdBJ5FNLzMy0yQ%40mail.gmail.com
<https://groups.google.com/d/msgid/elasticsearch/CA%2BhvuXc%2BuQWV5ySXa0407rHCmafEef4x4OU0qdBJ5FNLzMy0yQ%40mail.gmail.com>.

 > For more options, visit https://groups.google.com/groups/opt_out
<https://groups.google.com/groups/opt_out>.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c1e1c3e8-f2fc-43a6-bb9c-526be656eb73%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/le73el%24pin%241%40ger.gmane.org.
For more options, visit https://groups.google.com/groups/opt_out.


(Ivan Beveridge) #12

Hi Attila,

I don't have any direct experience on this, but I'm sure I read that you
can migrate shards from one server to another by specifying some kind of
preference (in the elasticsearch.yml or for the index).


http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-cluster.html

This kind of action has been previously discussed regarding migrating
old/less-used indexes from fast servers to ones with slower disks.

With this in mind, would it not be easier to:

  • expand the existing 1-node cluster (eg another 3 high-spec servers)
  • ensure the existing index has at least 1 replica (or add a replica)
  • configure new indexes/shards to prefer the new servers
  • migrate the old shards to the new servers
  • shutdown old server

Anything connecting to the ES cluster will either have to point to a new
(loadbalanced?) IP pointing at the new nodes .... or can use the
multicast (discovery) group address and same cluster name without
changing anything.

Assuming I am correct (that there is a way to do this), you end up with
no affect to the ES-cluster service.

Cheers

Ivan

On 19/02/2014 16:26, Attila Bukor wrote:

Hi Tony,

Thank you for your response, reading this I'm pretty sure I'm going with
Yann's suggestion to simply backup & restore. Your first point confirms
that
this is a good idea anyway.

To answer your second point, both clusters are on the same /24 subnet, the
problem is that oldcluster is not designed to do that, we put a production
site's index on a server with twenty-odd GB free space and 4GB RAM
total, so
we need to move it to the dedicated cluster.

The other two points are great suggestions and thank you very much for
them,
but I think the backup & restore way would be more suitable for my case.

Thank you very much,
Attila

On 02/19/2014 04:50 PM, Tony Su wrote:> Hi,

My testing has done a little bit of what you're describing...

  • FWIW and I don't know why, when I re-insert the data from scratch
    instead of upgrading a cluster, I'm seeing substantial improvements in
    node general health. They seem faster, less latencies related to
    indexing (I assume there should be no diff searching, but ??). If a
    reason why I'm seeing this can't be found, then maybe what I'm seeing is
    specific to my setup and hardware.
  • I don't know that there should be any reason to name your cluster
    differently if the networks are completely isolated, but of course if
    there should be a mistake, the consequences could be serious.
  • I've been using various methods that display index activity, the
    easiest to use (and today work on both 0.90 and 1.0) are
    elasticsearch-head and elasticsearch-hq. I've also been toying around
    recently with simply probing each node using a curl command, which
    depending on how you deploy web tools likely has a lower load effect on
    resources (I'm currently running Apache on one node and when I use one
    or more of the above tools, has a significant and measurable load).
  • If you do an upgrade, at least the one time after the upgrade you may
    want to force your cluster to wait until all nodes in the cluster are
    active before restarting to minimize shard thrashing (if a node is not
    online yet, the cluster may discard the indices on that node and
    re-allocate from whatever copy already is active. Then, when the missing
    node finally joins, then the shards are re-balanced). One way to do that
    is setting the following to be the same as the total nodes in the cluster
    gateway.recovery_after_nodes
    HTH,
    Tony

On Wednesday, February 19, 2014 7:01:03 AM UTC-8, Attila Bukor wrote:

Hey everybody,

I have a question regarding the migration of the indices to a new
cluster
seamlessly. Let me first describe the situation:

I have a project which uses Elasticsearch since a few weeks ago.
This is our
first Elasticsearch project, and as we are satisfied with it, we
decided to
dedicate a cluster of 3 nodes to run Elasticsearch (lets call it
"newcluster"). It is up and running separately from our old 1-node
Elasticsearch installation ("oldcluster"). We want to migrate
everything to
newcluster, and turn oldcluster off. All 4 servers are running
Ubuntu Server,
Elasticsearch is installed from the official deb package,

oldcluster is

0.90.10 and newcluster 1.0.0.

I came up with the following solution, but I'm not sure it will work:

- Upgrade oldcluster to 1.0.0 (tested it in dev, should work)
- Backup the index (we have only one index right now) in case
something goes
    wrong.
- Change oldcluster's name to newcluster.
- Wait until the nodes synchronize themselves (how can I check if
they're in
    sync?)
- Shut down oldcluster.

Thank you very much,
Attila

--
Ivan Beveridge

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5308D0CB.3050609%40livejournalinc.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Tony Su) #13

FYI -
Within the last couple days, I attempted to do what you propose.
You may want to experiment in a test cluster emulating your topology, too.

My steps
Snapshot the cluster

Then I emulated a new cluster and

  • Set a new cluster name in cluster.name
  • Renamed the Data Directory with the new cluster name

I discovered that renaming a cluster already populated with data is no
trivial thing. Shards were still thrashing 2hrs later when I finally gave
up.

So, I changed courses slightly and decided to remove the existing data
thereby emulating a new cluster.
Since I intended to restore from the snapshots, I renamed cluster.name back
to the original and changed the data directory to be consistent with the
cluster name again.

Then, after re-registering the snapshot directory I did a snapshot restore
which was 99% successful.
I then deleted the data and restored again which this time was 100%
successful and the cluster is still green.

Note I did not try to change the cluster name again. If you want to do so,
you should do so as a separate step like I described so if there is an
issue it's easy to trace and address.

Or, I would strongly recommend Ivan's suggestion. IMO creating new replicas
is nearly trivial and doesn't put an extraordinary load on the cluster so
you could migrate the new data to your new nodes.

Tony

On Saturday, February 22, 2014 8:31:07 AM UTC-8, Ivan Beveridge wrote:

Hi Attila,

I don't have any direct experience on this, but I'm sure I read that you
can migrate shards from one server to another by specifying some kind of
preference (in the elasticsearch.yml or for the index).

http://blog.sematext.com/2012/05/29/elasticsearch-shard-placement-control/

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-cluster.html

This kind of action has been previously discussed regarding migrating
old/less-used indexes from fast servers to ones with slower disks.

With this in mind, would it not be easier to:

  • expand the existing 1-node cluster (eg another 3 high-spec servers)
  • ensure the existing index has at least 1 replica (or add a replica)
  • configure new indexes/shards to prefer the new servers
  • migrate the old shards to the new servers
  • shutdown old server

Anything connecting to the ES cluster will either have to point to a new
(loadbalanced?) IP pointing at the new nodes .... or can use the
multicast (discovery) group address and same cluster name without
changing anything.

Assuming I am correct (that there is a way to do this), you end up with
no affect to the ES-cluster service.

Cheers

Ivan

On 19/02/2014 16:26, Attila Bukor wrote:

Hi Tony,

Thank you for your response, reading this I'm pretty sure I'm going with
Yann's suggestion to simply backup & restore. Your first point confirms
that
this is a good idea anyway.

To answer your second point, both clusters are on the same /24 subnet,
the
problem is that oldcluster is not designed to do that, we put a
production
site's index on a server with twenty-odd GB free space and 4GB RAM
total, so
we need to move it to the dedicated cluster.

The other two points are great suggestions and thank you very much for
them,
but I think the backup & restore way would be more suitable for my case.

Thank you very much,
Attila

On 02/19/2014 04:50 PM, Tony Su wrote:> Hi,

My testing has done a little bit of what you're describing...

  • FWIW and I don't know why, when I re-insert the data from scratch
    instead of upgrading a cluster, I'm seeing substantial improvements in
    node general health. They seem faster, less latencies related to
    indexing (I assume there should be no diff searching, but ??). If a
    reason why I'm seeing this can't be found, then maybe what I'm seeing
    is

specific to my setup and hardware.

  • I don't know that there should be any reason to name your cluster
    differently if the networks are completely isolated, but of course if
    there should be a mistake, the consequences could be serious.
  • I've been using various methods that display index activity, the
    easiest to use (and today work on both 0.90 and 1.0) are
    elasticsearch-head and elasticsearch-hq. I've also been toying around
    recently with simply probing each node using a curl command, which
    depending on how you deploy web tools likely has a lower load effect on
    resources (I'm currently running Apache on one node and when I use one
    or more of the above tools, has a significant and measurable load).
  • If you do an upgrade, at least the one time after the upgrade you may
    want to force your cluster to wait until all nodes in the cluster are
    active before restarting to minimize shard thrashing (if a node is not
    online yet, the cluster may discard the indices on that node and
    re-allocate from whatever copy already is active. Then, when the
    missing

node finally joins, then the shards are re-balanced). One way to do
that

is setting the following to be the same as the total nodes in the
cluster

gateway.recovery_after_nodes
HTH,
Tony

On Wednesday, February 19, 2014 7:01:03 AM UTC-8, Attila Bukor wrote:

Hey everybody, 

I have a question regarding the migration of the indices to a new 
cluster 
seamlessly. Let me first describe the situation: 

I have a project which uses Elasticsearch since a few weeks ago. 
This is our 
first Elasticsearch project, and as we are satisfied with it, we 
decided to 
dedicate a cluster of 3 nodes to run Elasticsearch (lets call it 
"newcluster"). It is up and running separately from our old 1-node 
Elasticsearch installation ("oldcluster"). We want to migrate 
everything to 
newcluster, and turn oldcluster off. All 4 servers are running 
Ubuntu Server, 
Elasticsearch is installed from the official deb package, 

oldcluster is

0.90.10 and newcluster 1.0.0. 

I came up with the following solution, but I'm not sure it will 

work:

- Upgrade oldcluster to 1.0.0 (tested it in dev, should work) 
- Backup the index (we have only one index right now) in case 
something goes 
    wrong. 
- Change oldcluster's name to newcluster. 
- Wait until the nodes synchronize themselves (how can I check if 
they're in 
    sync?) 
- Shut down oldcluster. 

Thank you very much, 
Attila 

--
Ivan Beveridge

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/250bc8bc-5811-420c-964b-f7d5169551a6%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(system) #14