Backing Up ES


(Eugene Strokin) #1

Hello, I'm planning to use very nice script by Karussell


for backing up my ES.
But I have a cluster of currently 2 machines running the ES.
I want to make sure, that this is enough to copy the files just from one
machine, because after flush they are the same on both server boxes.
And to restore, I just need to restore files on one machine, run ES, and on
another I'd just run empty ES server with my custom mappings, and the
second machine would pick the data up from the first.
Am I correct, or my assumption is mistaken?

Thank you,
Eugene S.


(Shay Banon) #2

Yes, assuming you have 1 replica, and 2 machines, you only need to copy one machine over. And the restore process you mentioned is good.

On Tuesday, March 13, 2012 at 5:24 AM, Eugene Strokin wrote:

Hello, I'm planning to use very nice script by Karussell
https://gist.github.com/1074906
for backing up my ES.

But I have a cluster of currently 2 machines running the ES.
I want to make sure, that this is enough to copy the files just from one machine, because after flush they are the same on both server boxes.
And to restore, I just need to restore files on one machine, run ES, and on another I'd just run empty ES server with my custom mappings, and the second machine would pick the data up from the first.
Am I correct, or my assumption is mistaken?

Thank you,
Eugene S.


(Barsk) #3

Regarding backup. Since I am using ES sort of as a noSQL database it
will not be possible to reindex from scratch after updates has been done
to the documents.
Is it guaranteed that it will always be possible to import an index to a
newer version of ES without need of reindexing it?

Second question. How do you backup shards that are not on the local
node? I am not there yet, but I might be in future...

/Kristian

Shay Banon skrev 2012-03-14 13:05:

Yes, assuming you have 1 replica, and 2 machines, you only need to
copy one machine over. And the restore process you mentioned is good.

On Tuesday, March 13, 2012 at 5:24 AM, Eugene Strokin wrote:

Hello, I'm planning to use very nice script by Karussell
https://gist.github.com/1074906
for backing up my ES.
But I have a cluster of currently 2 machines running the ES.
I want to make sure, that this is enough to copy the files just from
one machine, because after flush they are the same on both server boxes.
And to restore, I just need to restore files on one machine, run ES,
and on another I'd just run empty ES server with my custom mappings,
and the second machine would pick the data up from the first.
Am I correct, or my assumption is mistaken?

Thank you,
Eugene S.

--
Med vänlig hälsning
Kristian Jörg

Devo IT AB
Tel: 054 - 22 14 58, 0709 - 15 83 42
E-post: kristian.jorg@devo.se
Webb: http://www.devo.se


(Shay Banon) #4

Yes, you will always be able to upgrade to a newer version without needing
to reindex. A multi node cluster backup simplest option is to backup each
node data location.

On Mon, Mar 19, 2012 at 11:13 AM, Kristian Jörg krjg@devo.se wrote:

Regarding backup. Since I am using ES sort of as a noSQL database it will
not be possible to reindex from scratch after updates has been done to the
documents.
Is it guaranteed that it will always be possible to import an index to a
newer version of ES without need of reindexing it?

Second question. How do you backup shards that are not on the local node?
I am not there yet, but I might be in future...

/Kristian

Shay Banon skrev 2012-03-14 13:05:

Yes, assuming you have 1 replica, and 2 machines, you only need to copy

one machine over. And the restore process you mentioned is good.

On Tuesday, March 13, 2012 at 5:24 AM, Eugene Strokin wrote:

Hello, I'm planning to use very nice script by Karussell

https://gist.github.com/**1074906 https://gist.github.com/1074906
for backing up my ES.
But I have a cluster of currently 2 machines running the ES.
I want to make sure, that this is enough to copy the files just from one
machine, because after flush they are the same on both server boxes.
And to restore, I just need to restore files on one machine, run ES, and
on another I'd just run empty ES server with my custom mappings, and the
second machine would pick the data up from the first.
Am I correct, or my assumption is mistaken?

Thank you,
Eugene S.

--
Med vänlig hälsning
Kristian Jörg

Devo IT AB
Tel: 054 - 22 14 58, 0709 - 15 83 42
E-post: kristian.jorg@devo.se
Webb: http://www.devo.se


(Barsk) #5
Ok, but do one still need to temporarily disable flush as the script
does, on each node,  prior to backup? Or is the setting "global" for
the cluster?
-- 
Med vänlig hälsning
Kristian Jörg

Devo IT AB
Tel: 054 - 22 14 58, 0709 - 15 83 42
E-post: kristian.jorg@devo.se
Webb: http://www.devo.se
Shay Banon skrev 2012-03-20 11:41:
<blockquote cite="mid:CALzs+uxjRFMN4LwPWkTxDhyF1KtGVnNuWQdP_EGmOTfurs3aSw@mail.gmail.com" type="cite">Yes, you will always be able to upgrade to a newer
    version without needing to reindex. A multi node cluster backup
    simplest option is to backup each node data location.

On Mon, Mar 19, 2012 at 11:13 AM,
Kristian Jörg <krjg@devo.se>
wrote:

Regarding backup. Since I am using ES sort of as a noSQL database it will not be possible to reindex from scratch after updates has been done to the documents.
        Is it guaranteed that it will always be possible to import
        an index to a newer version of ES without need of reindexing
        it?




        Second question. How do you backup shards that are not on
        the local node? I am not there yet, but I might be in
        future...




        /Kristian




        Shay Banon skrev 2012-03-14 13:05:
Yes, assuming you have 1 replica, and 2 machines, you only need to copy one machine over. And the restore process you mentioned is good.
              On Tuesday, March 13, 2012 at 5:24 AM, Eugene Strokin
              wrote:
Hello, I'm planning to use very nice script by Karussell

https://gist.github.com/1074906

                for backing up my ES.


                But I have a cluster of currently 2 machines running
                the ES.


                I want to make sure, that this is enough to copy the
                files just from one machine, because after flush
                they are the same on both server boxes.


                And to restore, I just need to restore files on one
                machine, run ES, and on another I'd just run empty
                ES server with my custom mappings, and the second
                machine would pick the data up from the first.


                Am I correct, or my assumption is mistaken?




                Thank you,


                Eugene S.
            -- 


            Med vänlig hälsning


            Kristian Jörg




            Devo IT AB


            Tel: 054 - 22 14 58, 0709 - 15 83 42


            E-post: <a moz-do-not-send="true" href="mailto:kristian.jorg@devo.se" target="_blank">kristian.jorg@devo.se</a>


            Webb: <a moz-do-not-send="true" href="http://www.devo.se" target="_blank">http://www.devo.se</a>

      

(Shay Banon) #6

In the script, the update setting is global and applied to all indices. So
you need to execute it once and it will apply to the whole cluster.

On Tue, Mar 20, 2012 at 3:52 PM, Kristian Jörg krjg@devo.se wrote:

Ok, but do one still need to temporarily disable flush as the script
does, on each node, prior to backup? Or is the setting "global" for the
cluster?

--
Med vänlig hälsning
Kristian Jörg

Devo IT AB
Tel: 054 - 22 14 58, 0709 - 15 83 42
E-post: kristian.jorg@devo.se
Webb: http://www.devo.se

Shay Banon skrev 2012-03-20 11:41:

Yes, you will always be able to upgrade to a newer version without needing
to reindex. A multi node cluster backup simplest option is to backup each
node data location.

On Mon, Mar 19, 2012 at 11:13 AM, Kristian Jörg krjg@devo.se wrote:

Regarding backup. Since I am using ES sort of as a noSQL database it will
not be possible to reindex from scratch after updates has been done to the
documents.
Is it guaranteed that it will always be possible to import an index to a
newer version of ES without need of reindexing it?

Second question. How do you backup shards that are not on the local node?
I am not there yet, but I might be in future...

/Kristian

Shay Banon skrev 2012-03-14 13:05:

Yes, assuming you have 1 replica, and 2 machines, you only need to copy

one machine over. And the restore process you mentioned is good.

On Tuesday, March 13, 2012 at 5:24 AM, Eugene Strokin wrote:

Hello, I'm planning to use very nice script by Karussell

https://gist.github.com/1074906
for backing up my ES.
But I have a cluster of currently 2 machines running the ES.
I want to make sure, that this is enough to copy the files just from
one machine, because after flush they are the same on both server boxes.
And to restore, I just need to restore files on one machine, run ES,
and on another I'd just run empty ES server with my custom mappings, and
the second machine would pick the data up from the first.
Am I correct, or my assumption is mistaken?

Thank you,
Eugene S.

--
Med vänlig hälsning
Kristian Jörg

Devo IT AB
Tel: 054 - 22 14 58, 0709 - 15 83 42
E-post: kristian.jorg@devo.se
Webb: http://www.devo.se


(Barsk) #7
Yes, that is what I thought. 




Thanx for all help!


/Kristian






Shay Banon skrev 2012-03-20 21:17:
<blockquote cite="mid:CALzs+uwB6U+fGP=hzd8B+KZnPX9yUwmHNPU9tTz3RLx4HDHcOg@mail.gmail.com" type="cite">In the script, the update setting is global and
    applied to all indices. So you need to execute it once and it
    will apply to the whole cluster.

On Tue, Mar 20, 2012 at 3:52 PM,
Kristian Jörg <krjg@devo.se>
wrote:

Ok, but do one still need to temporarily disable flush as the script does, on each node, prior to backup? Or is the setting "global" for the cluster?
-- 
Med vänlig hälsning
Kristian Jörg

Devo IT AB
Tel: 054 - 22 14 58, 0709 - 15 83 42
E-post: kristian.jorg@devo.se
Webb: http://www.devo.se
          Shay Banon skrev 2012-03-20 11:41:
          <blockquote type="cite">Yes, you will always be able to
                  upgrade to a newer version without needing to
                  reindex. A multi node cluster backup simplest
                  option is to backup each node data location.

On Mon, Mar 19, 2012 at
11:13 AM, Kristian Jörg <krjg@devo.se>
wrote:

Regarding backup. Since I am using ES sort of as a noSQL database it will not be possible to reindex from scratch after updates has been done to the documents.
                      Is it guaranteed that it will always be
                      possible to import an index to a newer version
                      of ES without need of reindexing it?




                      Second question. How do you backup shards that
                      are not on the local node? I am not there yet,
                      but I might be in future...




                      /Kristian




                      Shay Banon skrev 2012-03-14 13:05:
Yes, assuming you have 1 replica, and 2 machines, you only need to copy one machine over. And the restore process you mentioned is good.
                            On Tuesday, March 13, 2012 at 5:24 AM,
                            Eugene Strokin wrote:
Hello, I'm planning to use very nice script by Karussell

https://gist.github.com/1074906

                              for backing up my ES.


                              But I have a cluster of currently 2
                              machines running the ES.


                              I want to make sure, that this is
                              enough to copy the files just from one
                              machine, because after flush they are
                              the same on both server boxes.


                              And to restore, I just need to restore
                              files on one machine, run ES, and on
                              another I'd just run empty ES server
                              with my custom mappings, and the
                              second machine would pick the data up
                              from the first.


                              Am I correct, or my assumption is
                              mistaken?




                              Thank you,


                              Eugene S.

--

                          Med vänlig hälsning


                          Kristian Jörg




                          Devo IT AB


                          Tel: 054 - 22 14 58, 0709 - 15 83 42


                          E-post: <a moz-do-not-send="true" href="mailto:kristian.jorg@devo.se" target="_blank">kristian.jorg@devo.se</a>


                          Webb: <a moz-do-not-send="true" href="http://www.devo.se" target="_blank">http://www.devo.se</a>
-- 
Med vänlig hälsning
Kristian Jörg

Devo IT AB
Tel: 054 - 22 14 58, 0709 - 15 83 42
E-post: kristian.jorg@devo.se
Webb: http://www.devo.se

(Frederic) #8

Hi Kimchy,

Based on what you said, I understand that f I needed to restore backed up
data, I would only need to replace current data directory with its copy,
shutting down the cluster first I guess.

So, lets suppose I want to recover the data but one of the servers fails
and it can't run again, but I can recover the data (stored in a remote
iSCSI in this case). I will only need to replace that server with a new
one, and copy/mount the data to the new server, to always have 1 node per
data copy right?

Thanks,

On Tuesday, 20 March 2012 07:41:31 UTC-3, kimchy wrote:

Yes, you will always be able to upgrade to a newer version without needing
to reindex. A multi node cluster backup simplest option is to backup each
node data location.


(Shay Banon) #9

Yes, you will just need to have the nodes pointing to the backed up data
location of each node. Then fire back the cluster.

On Wed, Mar 21, 2012 at 5:21 PM, Frederic focampo.br@gmail.com wrote:

Hi Kimchy,

Based on what you said, I understand that f I needed to restore backed up
data, I would only need to replace current data directory with its copy,
shutting down the cluster first I guess.

So, lets suppose I want to recover the data but one of the servers fails
and it can't run again, but I can recover the data (stored in a remote
iSCSI in this case). I will only need to replace that server with a new
one, and copy/mount the data to the new server, to always have 1 node per
data copy right?

Thanks,

On Tuesday, 20 March 2012 07:41:31 UTC-3, kimchy wrote:

Yes, you will always be able to upgrade to a newer version without
needing to reindex. A multi node cluster backup simplest option is to
backup each node data location.


(system) #10