How to add ec2 s3 or other gateway after index is created?

Elastic Stack Elasticsearch
1

hi,
i have a running ES cluster (of 1 node only) on ec2 with lots of data on
it.. this is without ec2 plugin and any gateway. now i want to configure a
s3 gateway on top of it. but if i follow the steps in the tutorial, it ends
up deleting my index! how do i get around this problem? is it even possible?

i took a copy of the data as a backup and hence i am able to retry multiple
times by restoring the data back..

thanks

ps - i followed the steps in the tutorial with a new cluster and it works
perfectly. also, for my experiments, i am renaming the cluster by simply
renaming the cluster name dir in data dir.

2

hi,
i tried different routes and still not able to get it to work. any ideas?
i even tried copying the whole data dir structure on s3 and using the same
name as cluster name and yet didnt work.. ES would empty out the cluster
name dir.

thanks

On Tue, Jul 17, 2012 at 4:10 PM, T Vinod Gupta tvinod@readypulse.comwrote:

hi,
i have a running ES cluster (of 1 node only) on ec2 with lots of data on
it.. this is without ec2 plugin and any gateway. now i want to configure a
s3 gateway on top of it. but if i follow the steps in the tutorial, it ends
up deleting my index! how do i get around this problem? is it even possible?

i took a copy of the data as a backup and hence i am able to retry
multiple times by restoring the data back..

thanks

ps - i followed the steps in the tutorial with a new cluster and it works
perfectly. also, for my experiments, i am renaming the cluster by simply
renaming the cluster name dir in data dir.

3

Its recommended to use the default local gateway even on AWS. No need to
move to s3 gateway really.

On Wed, Jul 18, 2012 at 12:21 PM, T Vinod Gupta tvinod@readypulse.comwrote:

hi,
i tried different routes and still not able to get it to work. any ideas?
i even tried copying the whole data dir structure on s3 and using the same
name as cluster name and yet didnt work.. ES would empty out the cluster
name dir.

thanks

On Tue, Jul 17, 2012 at 4:10 PM, T Vinod Gupta tvinod@readypulse.comwrote:

hi,
i have a running ES cluster (of 1 node only) on ec2 with lots of data on
it.. this is without ec2 plugin and any gateway. now i want to configure a
s3 gateway on top of it. but if i follow the steps in the tutorial, it ends
up deleting my index! how do i get around this problem? is it even possible?

i took a copy of the data as a backup and hence i am able to retry
multiple times by restoring the data back..

thanks

ps - i followed the steps in the tutorial with a new cluster and it works
perfectly. also, for my experiments, i am renaming the cluster by simply
renaming the cluster name dir in data dir.

4

just to clarify, local default gateway is what you specify in "path.data"
setting right?

what i want to achieve from s3 is to handle a scenario where my node that
uses local ephemeral disk as local data store and then due to some outage
or crash, the local data is lost/corrupted. i should be easily able to
create another instance and point it to s3 and then it restores all the
data locally on that machine and is able to serve data and index data..

is that clear?

thanks

On Wed, Jul 18, 2012 at 1:41 PM, Shay Banon kimchy@gmail.com wrote:

Its recommended to use the default local gateway even on AWS. No need to
move to s3 gateway really.

On Wed, Jul 18, 2012 at 12:21 PM, T Vinod Gupta tvinod@readypulse.comwrote:

hi,
i tried different routes and still not able to get it to work. any ideas?
i even tried copying the whole data dir structure on s3 and using the
same name as cluster name and yet didnt work.. ES would empty out the
cluster name dir.

thanks

On Tue, Jul 17, 2012 at 4:10 PM, T Vinod Gupta tvinod@readypulse.comwrote:

hi,
i have a running ES cluster (of 1 node only) on ec2 with lots of data on
it.. this is without ec2 plugin and any gateway. now i want to configure a
s3 gateway on top of it. but if i follow the steps in the tutorial, it ends
up deleting my index! how do i get around this problem? is it even possible?

i took a copy of the data as a backup and hence i am able to retry
multiple times by restoring the data back..

thanks

ps - i followed the steps in the tutorial with a new cluster and it
works perfectly. also, for my experiments, i am renaming the cluster by
simply renaming the cluster name dir in data dir.

5

It is clear, but you can also simply store it on EBS, or have more nodes
with more copies of your data.

On Wed, Jul 18, 2012 at 1:47 PM, T Vinod Gupta tvinod@readypulse.comwrote:

just to clarify, local default gateway is what you specify in "path.data"
setting right?

what i want to achieve from s3 is to handle a scenario where my node that
uses local ephemeral disk as local data store and then due to some outage
or crash, the local data is lost/corrupted. i should be easily able to
create another instance and point it to s3 and then it restores all the
data locally on that machine and is able to serve data and index data..

is that clear?

thanks

On Wed, Jul 18, 2012 at 1:41 PM, Shay Banon kimchy@gmail.com wrote:

Its recommended to use the default local gateway even on AWS. No need to
move to s3 gateway really.

On Wed, Jul 18, 2012 at 12:21 PM, T Vinod Gupta tvinod@readypulse.comwrote:

hi,
i tried different routes and still not able to get it to work. any ideas?
i even tried copying the whole data dir structure on s3 and using the
same name as cluster name and yet didnt work.. ES would empty out the
cluster name dir.

thanks

On Tue, Jul 17, 2012 at 4:10 PM, T Vinod Gupta tvinod@readypulse.comwrote:

hi,
i have a running ES cluster (of 1 node only) on ec2 with lots of data
on it.. this is without ec2 plugin and any gateway. now i want to configure
a s3 gateway on top of it. but if i follow the steps in the tutorial, it
ends up deleting my index! how do i get around this problem? is it even
possible?

i took a copy of the data as a backup and hence i am able to retry
multiple times by restoring the data back..

thanks

ps - i followed the steps in the tutorial with a new cluster and it
works perfectly. also, for my experiments, i am renaming the cluster by
simply renaming the cluster name dir in data dir.

6

:slight_smile:
i know the approach you mentioned would work.. but i am exploring s3 option
to save on cost. my current traffic doesnt really warrant 2 nodes. so i
want to be on 1 node (to save money). and i am also not inclined on ebs
because it would cost more and moreover, if i have 1 node, then i have to
worry about taking periodic snapshots to keep a backup.
so to save all this trouble and cost, i am interested in s3.

thanks

On Wed, Jul 18, 2012 at 1:52 PM, Shay Banon kimchy@gmail.com wrote:

It is clear, but you can also simply store it on EBS, or have more nodes
with more copies of your data.

On Wed, Jul 18, 2012 at 1:47 PM, T Vinod Gupta tvinod@readypulse.comwrote:

just to clarify, local default gateway is what you specify in "path.data"
setting right?

what i want to achieve from s3 is to handle a scenario where my node that
uses local ephemeral disk as local data store and then due to some outage
or crash, the local data is lost/corrupted. i should be easily able to
create another instance and point it to s3 and then it restores all the
data locally on that machine and is able to serve data and index data..

is that clear?

thanks

On Wed, Jul 18, 2012 at 1:41 PM, Shay Banon kimchy@gmail.com wrote:

Its recommended to use the default local gateway even on AWS. No need to
move to s3 gateway really.

On Wed, Jul 18, 2012 at 12:21 PM, T Vinod Gupta tvinod@readypulse.comwrote:

hi,
i tried different routes and still not able to get it to work. any
ideas?
i even tried copying the whole data dir structure on s3 and using the
same name as cluster name and yet didnt work.. ES would empty out the
cluster name dir.

thanks

On Tue, Jul 17, 2012 at 4:10 PM, T Vinod Gupta tvinod@readypulse.comwrote:

hi,
i have a running ES cluster (of 1 node only) on ec2 with lots of data
on it.. this is without ec2 plugin and any gateway. now i want to configure
a s3 gateway on top of it. but if i follow the steps in the tutorial, it
ends up deleting my index! how do i get around this problem? is it even
possible?

i took a copy of the data as a backup and hence i am able to retry
multiple times by restoring the data back..

thanks

ps - i followed the steps in the tutorial with a new cluster and it
works perfectly. also, for my experiments, i am renaming the cluster by
simply renaming the cluster name dir in data dir.

7

another reason for not using EBS is that it is slower than local disk.
thanks

On Wed, Jul 18, 2012 at 2:00 PM, T Vinod Gupta tvinod@readypulse.comwrote:

:slight_smile:
i know the approach you mentioned would work.. but i am exploring s3
option to save on cost. my current traffic doesnt really warrant 2 nodes.
so i want to be on 1 node (to save money). and i am also not inclined on
ebs because it would cost more and moreover, if i have 1 node, then i have
to worry about taking periodic snapshots to keep a backup.
so to save all this trouble and cost, i am interested in s3.

thanks

On Wed, Jul 18, 2012 at 1:52 PM, Shay Banon kimchy@gmail.com wrote:

It is clear, but you can also simply store it on EBS, or have more nodes
with more copies of your data.

On Wed, Jul 18, 2012 at 1:47 PM, T Vinod Gupta tvinod@readypulse.comwrote:

just to clarify, local default gateway is what you specify in
"path.data" setting right?

what i want to achieve from s3 is to handle a scenario where my node
that uses local ephemeral disk as local data store and then due to some
outage or crash, the local data is lost/corrupted. i should be easily able
to create another instance and point it to s3 and then it restores all the
data locally on that machine and is able to serve data and index data..

is that clear?

thanks

On Wed, Jul 18, 2012 at 1:41 PM, Shay Banon kimchy@gmail.com wrote:

Its recommended to use the default local gateway even on AWS. No need
to move to s3 gateway really.

On Wed, Jul 18, 2012 at 12:21 PM, T Vinod Gupta tvinod@readypulse.comwrote:

hi,
i tried different routes and still not able to get it to work. any
ideas?
i even tried copying the whole data dir structure on s3 and using the
same name as cluster name and yet didnt work.. ES would empty out the
cluster name dir.

thanks

On Tue, Jul 17, 2012 at 4:10 PM, T Vinod Gupta tvinod@readypulse.comwrote:

hi,
i have a running ES cluster (of 1 node only) on ec2 with lots of data
on it.. this is without ec2 plugin and any gateway. now i want to configure
a s3 gateway on top of it. but if i follow the steps in the tutorial, it
ends up deleting my index! how do i get around this problem? is it even
possible?

i took a copy of the data as a backup and hence i am able to retry
multiple times by restoring the data back..

thanks

ps - i followed the steps in the tutorial with a new cluster and it
works perfectly. also, for my experiments, i am renaming the cluster by
simply renaming the cluster name dir in data dir.

8

T Vinod Gupta wrote:

another reason for not using EBS is that it is slower than local disk.

This is not always true, especially if you stripe your EBS volumes
appropriately. We've also had private talks with AWS about their
next gen EBS offering which will even further improve IOPS.

-Drew