Advice needed: migrating off the s3 gateway


(Paul Sanwald) #1

Hi Everyone,
I know the S3 gateway is deprecated, and I'm looking to move away from
using it. but, because of the way I deploy and run elasticsearch, I'm not
sure the best way to do this; and was hoping to get some advice from the
group. I'll try to give a detailed explanation of my deployment, bear with
me.

I have a web service that uses elasticsearch as its datastore. This web
service runs using amazon's elasticbeanstalk infrastructure; this allows me
to autoscale and do some other nice things around managing deployments, but
it does mean that instances get swapped around as AWS needs to, much like
heroku, I can't rely on my instances having static IPs, and I can't rely on
persistent local storage. When I set up my service (which is a java web
service running under tomcat), instead of running elasticsearch as a
service, I used a plugin to start up the node directly in my webservice.
This has worked very well for us, as I just use the S3 gateway, and when I
restart my service, the data is just read off of S3.

The documentation says I should move to the local gateway, but, it doesn't
seem like this will work with the way I have my deployment architected: I
don't have persistent local storage on elasticbeanstalk, unless I do
something fancy using an AMI, which is a possibility.

Do I just need to move off of elastic beanstalk entirely, and use standard
EC2 instances with manual deployment, if I want to use the local storage
gateway? Is there another gateway that will give me shared, non-local
storage?

thanks very much for reading, and for any advice you might have.

best,

--paul

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Oliver K) #2

Any news on this? Have you got any good solution?

I am planning the production release of an webapp based on php, mysql (AWS
RDS) and elasticsearch. I would really prefer to use the same beanstalk
cluster for the webapp and elasticsearch. Running another additional
cluster for the elasticsearch nodes sounds like more administrative work
afterwards and also mean more costs.

What is the best practice structure for using Beanstalk on Amazon EC2 and
also having a good backup solution if an instance goes down...

Am Sonntag, 6. Oktober 2013 13:33:23 UTC+2 schrieb Paul Sanwald:

Hi Everyone,
I know the S3 gateway is deprecated, and I'm looking to move away from
using it. but, because of the way I deploy and run elasticsearch, I'm not
sure the best way to do this; and was hoping to get some advice from the
group. I'll try to give a detailed explanation of my deployment, bear with
me.

I have a web service that uses elasticsearch as its datastore. This web
service runs using amazon's elasticbeanstalk infrastructure; this allows me
to autoscale and do some other nice things around managing deployments, but
it does mean that instances get swapped around as AWS needs to, much like
heroku, I can't rely on my instances having static IPs, and I can't rely on
persistent local storage. When I set up my service (which is a java web
service running under tomcat), instead of running elasticsearch as a
service, I used a plugin to start up the node directly in my webservice.
This has worked very well for us, as I just use the S3 gateway, and when I
restart my service, the data is just read off of S3.

The documentation says I should move to the local gateway, but, it doesn't
seem like this will work with the way I have my deployment architected: I
don't have persistent local storage on elasticbeanstalk, unless I do
something fancy using an AMI, which is a possibility.

Do I just need to move off of elastic beanstalk entirely, and use standard
EC2 instances with manual deployment, if I want to use the local storage
gateway? Is there another gateway that will give me shared, non-local
storage?

thanks very much for reading, and for any advice you might have.

best,

--paul

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/54a46941-a047-45bd-bd1b-846f9c8dd81c%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(system) #3