Hey guys,
I've found these 2 threads which seem to be quite helpful in my case:
http://elasticsearch-users.115913.n3.nabble.com/Copying-and-renaming-a-cluster-td2840624.html
http://elasticsearch-users.115913.n3.nabble.com/Changing-cluster-name-on-2-node-cluster-causes-IndexMissingException-td4035300.html
According to this, it should be possible to copy the whole my_backup
directory to newcluster and then restore.
Also, by setting "include_global_state" to false, I think there should be no
problem with a different cluster name and config. Of course the config is
the same newcluster-wide - except for the node names of course -, but it
differs from the one node in oldcluster.
Can somebody confirm or contradict this please?
Regards,
Attila
On 02/20/2014 05:34 PM, Tony Su wrote:
Although it might not be clear in the documentation,
I think the strong inference is that you need to set up the new cluster
with the same requirements for creating the snapshots in the first place.
This means...
- Shared File Repository. You need to configure every new node with the
same access (maybe even same path) as the old cluster.
- I think you need to create your new cluster using the same exact same
cluster and node configuration. This probably means copying your old
elasticsearch.yml on each node to the new corresponding node.
- After doing some snapshots myself, I can see how all the data is
stored in the backup, so although you copy data to your new
cluster, I can't see how it would be any benefit. Assuming you intend to
restore the full backup and not individual indices, everything will be
over-written.
I'm also going to guess (maybe needs verification) there may be problems
naming your cluster something different (needs verification but it makes
sense).
So, I would guess that after you do your snapshots you should just
shutdown your cluster unless it's a Production cluster. If you run both
clusters simultaneously, then they need to be physically isolated. Then,
after either switching or connecting your shared storage where the
snapshots are located, execute a restore to the new cluster. If the
original cluster is still running, then you can always make a snapshot
of the changes that happened after the data and state has been restored
to the new cluster and migrate the new, incremental data the same way.
IMO,
Based on what I've observed so far...
Like doing anything else this new, IMO it would be prudent to plan for
incremental steps in the whole process which can be back-tracked if you
run into something unexpected.
Tony
On Thursday, February 20, 2014 4:42:11 AM UTC-8, Attila Bukor wrote:
Hi,
I've read this one, but it's not clear to me how to do it on a
different
machine.
If I do a backup on oldcluster with
curl -XPUT oldcluster:9200/_snapshot/my_backup/snapshot_1 (assuming
I've
already created the my_backup repo), there will be metadata-snapshot_1,
snapshot-snapshot_1 and some other files under
/var/lib/elasticsearch/my_backup or some other place which is
configured.
Should I just tar the whole directory, scp it to node1 on newcluster
and
call
curl -XPOST
newcluster-node1:9200/_snapshot/my_backup/snapshot_1/restore ?
I don't really want to break things and this situation is not really
explained in the docs.
Cheers,
Attila
On 02/20/2014 01:14 PM, Yann Barraud wrote:
>
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html#_restore
<http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html#_restore>
> :-)
>
>
> Cordialement,
> Yann Barraud
>
>
> 2014-02-20 12:03 GMT+01:00 Attila Bukor
> <r1pp3...@w4it.eu <javascript:>
> <mailto:r1pp3...@w4it.eu <javascript:>>>:
>
> Hi Tony,
>
> The problem is the hardware, that is why we need to move it
to the new
> cluster.
>
> Yann, how can I restore a snapshot on a different machine?
>
> Cheers,
> Attila
>
>
> On 02/19/2014 07:00 PM, Tony Su wrote:
>
> I was wondering which backup/restore method you intended
to use.
>
> Snapshot/restore requires 1.0.
> If you intend to upgrade your 0.9.x cluster to 1.0 first, it
> looks like
> possibly a really good option (I intend to start testing
> snapshot/restore sometime soon. I've already created
snapshots
> and am
> impressed how it seems to take roughly the same amount to
create a
> snapshot no matter how much data is being backed up,
probably
> within reason)
>
> If your chosen path requires upgrading the existing
cluster to 1.0
> anyway, why migrate to the new cluster?
>
> Tony
>
>
>
> On Wednesday, February 19, 2014 8:16:02 AM UTC-8, Attila
Bukor
> wrote:
>
> Hi Yann,
>
> Thank you for your response, you saved me from a
*lot* of
> headache.
> So the
> revised flow:
>
> - Upgrade oldcluster to 1.0.0.
> - Implement a logger in the application which dumps all
> index API calls
> into a file.
> - Backup my index on oldcluster following these docs:
>
>
http://www.elasticsearch.org/__guide/en/elasticsearch/__reference/master/modules-__snapshots.html
<http://www.elasticsearch.org/__guide/en/elasticsearch/__reference/master/modules-__snapshots.html>
>
<http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-snapshots.html
<http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-snapshots.html>>
>
>
<http://www.elasticsearch.org/__guide/en/elasticsearch/__reference/master/modules-__snapshots.html
<http://www.elasticsearch.org/__guide/en/elasticsearch/__reference/master/modules-__snapshots.html>
>
<http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-snapshots.html
<http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-snapshots.html>>>
>
> - Restore the backed up index on newcluster
following the
> same docs.
> - Run the log from after the last _id in the snapshot.
> Repeat until
> they're the same.
> - Change the IP to newcluster in the application.
> - Repeat step 5 one more time.
> - Remove logging, delete logs.
> - Shut down oldcluster.
>
> In this case, the switch should not be noticed by
anyone
> and the data
> integrity is guaranteed, right?
>
> Cheers,
> Attila
>
> On Wed, 19 Feb 2014, Yann Barraud wrote:
>
> > Hi,
> >
> > If I get you well, you'll face an issue. Renaming
> oldcluster will
> make it "forget" previous indexes. (Take a look at data
> you'll find a
> dir named "oldcluster").
> >
> > The step you're missing is to backup data fro
oldcluster
> > Then shtudown, rename
> > Then restore...
> >
> > Or even shutdown and restore on new cluster...
> >
> > Cheers,
> > Yann
> >
> >
> >
> > Le mercredi 19 février 2014 16:01:03 UTC+1,
Attila Bukor
> a écrit :
> > Hey everybody,
> >
> > I have a question regarding the migration
of the
> indices to a
> new cluster
> > seamlessly. Let me first describe the
situation:
> >
> > I have a project which uses Elasticsearch
since a
> few weeks
> ago. This is our
> > first Elasticsearch project, and as we are
> satisfied with it,
> we decided to
> > dedicate a cluster of 3 nodes to run
Elasticsearch
> (lets
> call it
> > "newcluster"). It is up and running separately
> from our old
> 1-node
> > Elasticsearch installation ("oldcluster").
We want
> to migrate
> everything to
> > newcluster, and turn oldcluster off. All 4
servers
> are running
> Ubuntu Server,
> > Elasticsearch is installed from the
official deb
> package,
> oldcluster is
> > 0.90.10 and newcluster 1.0.0.
> >
> > I came up with the following solution, but
I'm not
> sure it
> will
> work:
> >
> > - Upgrade oldcluster to 1.0.0 (tested it in
dev,
> should work)
> > - Backup the index (we have only one index
right
> now) in case
> something goes
> > wrong.
> > - Change oldcluster's name to newcluster.
> > - Wait until the nodes synchronize
themselves (how
> can I check
> if they're in
> > sync?)
> > - Shut down oldcluster.
> >
> > Thank you very much,
> > Attila
> >
> >
> >
>
>
> --
> You received this message because you are subscribed to
the Google
> Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails
from
> it, send
>
> an email to
> elasticsearch+unsubscribe@__googlegroups.com
<http://googlegroups.com>
>
<mailto:elasticsearch%2Bunsubscribe@googlegroups.com
<javascript:>>.
> To view this discussion on the web visit
>
https://groups.google.com/d/__msgid/elasticsearch/f375faef-__2dd4-416d-8150-75afe2318f0c%__40googlegroups.com
<https://groups.google.com/d/__msgid/elasticsearch/f375faef-__2dd4-416d-8150-75afe2318f0c%__40googlegroups.com>
>
<https://groups.google.com/d/msgid/elasticsearch/f375faef-2dd4-416d-8150-75afe2318f0c%40googlegroups.com
<https://groups.google.com/d/msgid/elasticsearch/f375faef-2dd4-416d-8150-75afe2318f0c%40googlegroups.com>>.
> For more options, visit
> https://groups.google.com/__groups/opt_out
<https://groups.google.com/__groups/opt_out>
> <https://groups.google.com/groups/opt_out
<https://groups.google.com/groups/opt_out>>.
>
>
>
> --
> You received this message because you are subscribed to a
topic in
> the Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
>
https://groups.google.com/d/__topic/elasticsearch/__aOVumOxmsuQ/unsubscribe
<https://groups.google.com/d/__topic/elasticsearch/__aOVumOxmsuQ/unsubscribe>
>
<https://groups.google.com/d/topic/elasticsearch/aOVumOxmsuQ/unsubscribe
<https://groups.google.com/d/topic/elasticsearch/aOVumOxmsuQ/unsubscribe>>.
> To unsubscribe from this group and all its topics, send an
email to
> elasticsearch+unsubscribe@__googlegroups.com
<http://googlegroups.com>
>
<mailto:elasticsearch%2Bunsubscribe@googlegroups.com
<javascript:>>.
> To view this discussion on the web visit
>
https://groups.google.com/d/__msgid/elasticsearch/le4ne3%__24cg5%241%40ger.gmane.org
<https://groups.google.com/d/__msgid/elasticsearch/le4ne3%__24cg5%241%40ger.gmane.org>
>
<https://groups.google.com/d/msgid/elasticsearch/le4ne3%24cg5%241%40ger.gmane.org
<https://groups.google.com/d/msgid/elasticsearch/le4ne3%24cg5%241%40ger.gmane.org>>.
>
> For more options, visit
https://groups.google.com/__groups/opt_out
<https://groups.google.com/__groups/opt_out>
> <https://groups.google.com/groups/opt_out
<https://groups.google.com/groups/opt_out>>.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it,
send
> an email to
> elasticsearc...@googlegroups.com <javascript:>.
> To view this discussion on the web visit
>
https://groups.google.com/d/msgid/elasticsearch/CA%2BhvuXc%2BuQWV5ySXa0407rHCmafEef4x4OU0qdBJ5FNLzMy0yQ%40mail.gmail.com
<https://groups.google.com/d/msgid/elasticsearch/CA%2BhvuXc%2BuQWV5ySXa0407rHCmafEef4x4OU0qdBJ5FNLzMy0yQ%40mail.gmail.com>.
> For more options, visit https://groups.google.com/groups/opt_out
<https://groups.google.com/groups/opt_out>.
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c1e1c3e8-f2fc-43a6-bb9c-526be656eb73%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/le73el%24pin%241%40ger.gmane.org.
For more options, visit https://groups.google.com/groups/opt_out.
