We've begun deploying to AWS EC2. I've seen refrences in the group about
the S3 gateway and it being deprecated. That seems to be confirmed by
looking at the docs, which don't seem to list the S3 Gateway specifically
after 0.90.x.
We are also using the elasticsearch-cloud-aws plugins
https://github.com/elasticsearch/elasticsearch-cloud-aws, which does a
nice job at helping the auto discovery. It also shows settings for using S3.
After some reading my understanding is that the plugin is basically just
snapshots that are stored in S3. Is that understanding correct? Is this
much different from the original gateway?
That suggests that unless we take frequent snapshots we would run a risk of
data loss if the entire cluster wen't down (right now we are using instance
storage). Is that right?
Switching to EBS would give us better protection against data loss, since
the data is stored on a more permanent basis as well as improved recovery
after an entire cluster going down?
Are there any good guides on configuring this sort of setup with
cloudformation and templates and/or tying EBS volumes for ES use to
machines when a cluster is resurrected?
@matthias
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/415a7669-4c2a-4d3e-a960-67390c1197cf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Yes, you don't want to use anything other than local storage for
Elasticsearch. Not EBS and definitely not S3. You can use the
snapshot/restore API to continously backup to S3 and get all the data
protection you need.
--
Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Author of RavenDB in Action http://manning.com/synhershko/
On Tue, Oct 14, 2014 at 12:17 AM, Matthias Johnson opennomad@gmail.com
wrote:
We've begun deploying to AWS EC2. I've seen refrences in the group about
the S3 gateway and it being deprecated. That seems to be confirmed by
looking at the docs, which don't seem to list the S3 Gateway specifically
after 0.90.x.
We are also using the elasticsearch-cloud-aws plugins
https://github.com/elasticsearch/elasticsearch-cloud-aws, which does a
nice job at helping the auto discovery. It also shows settings for using S3.
After some reading my understanding is that the plugin is basically just
snapshots that are stored in S3. Is that understanding correct? Is this
much different from the original gateway?
That suggests that unless we take frequent snapshots we would run a risk
of data loss if the entire cluster wen't down (right now we are using
instance storage). Is that right?
Switching to EBS would give us better protection against data loss, since
the data is stored on a more permanent basis as well as improved recovery
after an entire cluster going down?
Are there any good guides on configuring this sort of setup with
cloudformation and templates and/or tying EBS volumes for ES use to
machines when a cluster is resurrected?
@matthias
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/415a7669-4c2a-4d3e-a960-67390c1197cf%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/415a7669-4c2a-4d3e-a960-67390c1197cf%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtPqGN7PDTQkoYcsYwfM_3bmVrEECwZcCiNqDsLsa9gqQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Or, if your use case allows for it, have a very well oiled rebuild process
(data included).
On 14/10/2014 8:36 am, "Itamar Syn-Hershko" itamar@code972.com wrote:
Yes, you don't want to use anything other than local storage for
Elasticsearch. Not EBS and definitely not S3. You can use the
snapshot/restore API to continously backup to S3 and get all the data
protection you need.
--
Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Author of RavenDB in Action http://manning.com/synhershko/
On Tue, Oct 14, 2014 at 12:17 AM, Matthias Johnson opennomad@gmail.com
wrote:
We've begun deploying to AWS EC2. I've seen refrences in the group about
the S3 gateway and it being deprecated. That seems to be confirmed by
looking at the docs, which don't seem to list the S3 Gateway specifically
after 0.90.x.
We are also using the elasticsearch-cloud-aws plugins
https://github.com/elasticsearch/elasticsearch-cloud-aws, which does a
nice job at helping the auto discovery. It also shows settings for using S3.
After some reading my understanding is that the plugin is basically just
snapshots that are stored in S3. Is that understanding correct? Is this
much different from the original gateway?
That suggests that unless we take frequent snapshots we would run a risk
of data loss if the entire cluster wen't down (right now we are using
instance storage). Is that right?
Switching to EBS would give us better protection against data loss, since
the data is stored on a more permanent basis as well as improved recovery
after an entire cluster going down?
Are there any good guides on configuring this sort of setup with
cloudformation and templates and/or tying EBS volumes for ES use to
machines when a cluster is resurrected?
@matthias
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/415a7669-4c2a-4d3e-a960-67390c1197cf%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/415a7669-4c2a-4d3e-a960-67390c1197cf%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtPqGN7PDTQkoYcsYwfM_3bmVrEECwZcCiNqDsLsa9gqQ%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtPqGN7PDTQkoYcsYwfM_3bmVrEECwZcCiNqDsLsa9gqQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CACj2-4%2BQqbg9_PF7i-timWB14_nrZ5asFj3%3DXxk5Wd8qabZvcw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.