AWS S3 snapshot copy

Elastic Stack Elasticsearch
1

Hello,
sometimes I want to grab index from production and run it locally.
On production it uses S3 Gateway. So, I wrote a little program which
copy all the files from production bucket to backup bucket, and
download the files from there to my local machine.
Copying from one bucket to another takes not much time.
But once I put those files under my local instance of ES it just
doesn't recover.

[2011-12-23 15:13:18,288][INFO ][cluster.service ] [Red
Ronin] new_master [Red Ronin][UbdeFcwVTG2beuTt-inuVw][i net[/192.168.0.4:9300]], reason: zen-disco-join (elected_as_master)
[2011-12-23 15:13:18,387][INFO ][discovery ] [Red
Ronin] my-es-cluster/UbdeFcwVTG2beuTt-inuVw
[2011-12-23 15:13:18,414][INFO ][gateway ] [Red
Ronin] recovered [0] indices into cluster_state

May be there is some official way to copy snapshots to run them
locally? Maybe I should just do something special do make it run?

Thanks in advance, any help is appreciated.

2

There isn't a way to copy the S3 data and place it into a "local" gateway
node(s).

On Fri, Dec 23, 2011 at 10:46 PM, Eugene Strokin eugene@strokin.infowrote:

Hello,
sometimes I want to grab index from production and run it locally.
On production it uses S3 Gateway. So, I wrote a little program which
copy all the files from production bucket to backup bucket, and
download the files from there to my local machine.
Copying from one bucket to another takes not much time.
But once I put those files under my local instance of ES it just
doesn't recover.

[2011-12-23 15:13:18,288][INFO ][cluster.service ] [Red
Ronin] new_master [Red Ronin][UbdeFcwVTG2beuTt-inuVw][i net[/192.168.0.4:9300]], reason: zen-disco-join (elected_as_master)
[2011-12-23 15:13:18,387][INFO ][discovery ] [Red
Ronin] my-es-cluster/UbdeFcwVTG2beuTt-inuVw
[2011-12-23 15:13:18,414][INFO ][gateway ] [Red
Ronin] recovered [0] indices into cluster_state

May be there is some official way to copy snapshots to run them
locally? Maybe I should just do something special do make it run?

Thanks in advance, any help is appreciated.

3

Oh, I c, what would be the recommended way to get a snapshot from AWS
EC/S3 Elasticsearch server?
Also, I just wander, if I want to make a daily backup of the index
files, would it be enough to just copy the bucket and restore it if
something went wrong?
But the real problem, is to get a data from the server, and use it
somewhere else
Thank u

On Dec 25, 11:32 am, Shay Banon kim...@gmail.com wrote:

There isn't a way to copy the S3 data and place it into a "local" gateway
node(s).

On Fri, Dec 23, 2011 at 10:46 PM, Eugene Strokin eug...@strokin.infowrote:

Hello,
sometimes I want to grab index from production and run it locally.
On production it uses S3 Gateway. So, I wrote a little program which
copy all the files from production bucket to backup bucket, and
download the files from there to my local machine.
Copying from one bucket to another takes not much time.
But once I put those files under my local instance of ES it just
doesn't recover.

[2011-12-23 15:13:18,288][INFO ][cluster.service ] [Red
Ronin] new_master [Red Ronin][UbdeFcwVTG2beuTt-inuVw][i net[/192.168.0.4:9300]], reason: zen-disco-join (elected_as_master)
[2011-12-23 15:13:18,387][INFO ][discovery ] [Red
Ronin] my-es-cluster/UbdeFcwVTG2beuTt-inuVw
[2011-12-23 15:13:18,414][INFO ][gateway ] [Red
Ronin] recovered [0] indices into cluster_state

May be there is some official way to copy snapshots to run them
locally? Maybe I should just do something special do make it run?

Thanks in advance, any help is appreciated.

4

You can start the other cluster on the other servers using s3 gateway as
well, pointing to the copy you made of your s3 data.

On Mon, Dec 26, 2011 at 7:03 AM, Eugene Strokin eugene@strokin.info wrote:

Oh, I c, what would be the recommended way to get a snapshot from AWS
EC/S3 Elasticsearch server?
Also, I just wander, if I want to make a daily backup of the index
files, would it be enough to just copy the bucket and restore it if
something went wrong?
But the real problem, is to get a data from the server, and use it
somewhere else
Thank u

On Dec 25, 11:32 am, Shay Banon kim...@gmail.com wrote:

There isn't a way to copy the S3 data and place it into a "local" gateway
node(s).

On Fri, Dec 23, 2011 at 10:46 PM, Eugene Strokin <eug...@strokin.info
wrote:

Hello,
sometimes I want to grab index from production and run it locally.
On production it uses S3 Gateway. So, I wrote a little program which
copy all the files from production bucket to backup bucket, and
download the files from there to my local machine.
Copying from one bucket to another takes not much time.
But once I put those files under my local instance of ES it just
doesn't recover.

[2011-12-23 15:13:18,288][INFO ][cluster.service ] [Red
Ronin] new_master [Red Ronin][UbdeFcwVTG2beuTt-inuVw][i net[/192.168.0.4:9300]], reason: zen-disco-join (elected_as_master)
[2011-12-23 15:13:18,387][INFO ][discovery ] [Red
Ronin] my-es-cluster/UbdeFcwVTG2beuTt-inuVw
[2011-12-23 15:13:18,414][INFO ][gateway ] [Red
Ronin] recovered [0] indices into cluster_state

May be there is some official way to copy snapshots to run them
locally? Maybe I should just do something special do make it run?

Thanks in advance, any help is appreciated.