Running Elasticsearch on AWS EC2

Hello,

I'm trying to run Elasticsearch on EC2. After watching the webinar on
running Elasticsearch on EC2, It was mentioned that the only working
solution is using ephemeral storage since EBS is not fast enough.
However, the current EC2 instance M3.large or even m3.xlarge come with very
small disks that won't be enough (40GB). I want to ship 25GB a day and
retain data for 30 days.

How do you recommend that I go about setting it up? Has anyone successfully
used EBS?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0d460d38-78a3-444b-9dd2-542f16015393%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

If you use EBS, then ask for provisioned IO.

HTH

David

Le 15 nov. 2014 à 09:01, Tomer Levy tomer.levy9@gmail.com a écrit :

Hello,

I'm trying to run Elasticsearch on EC2. After watching the webinar on running Elasticsearch on EC2, It was mentioned that the only working solution is using ephemeral storage since EBS is not fast enough.
However, the current EC2 instance M3.large or even m3.xlarge come with very small disks that won't be enough (40GB). I want to ship 25GB a day and retain data for 30 days.

How do you recommend that I go about setting it up? Has anyone successfully used EBS?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0d460d38-78a3-444b-9dd2-542f16015393%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6770DA70-03DE-4AB8-B69A-3A80456AE0D9%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

With the relatively recent transition of default EBS from magnetic to
SSD-backed storage with burstable IO, I suggest you test out your workloads
and see what meets your required performance levels. I've successfully run
ES on slow magentic-EBS when I couldn't justify the cost of PIOPS+EBS
optimized instances but your needs may be (read: probably are) different
than mine.

David

On Saturday, November 15, 2014 12:57:18 AM UTC-8, David Pilato wrote:

If you use EBS, then ask for provisioned IO.

HTH

David

Le 15 nov. 2014 à 09:01, Tomer Levy <tomer...@gmail.com <javascript:>> a
écrit :

Hello,

I'm trying to run Elasticsearch on EC2. After watching the webinar on
running Elasticsearch on EC2, It was mentioned that the only working
solution is using ephemeral storage since EBS is not fast enough.
However, the current EC2 instance M3.large or even m3.xlarge come with
very small disks that won't be enough (40GB). I want to ship 25GB a day and
retain data for 30 days.

How do you recommend that I go about setting it up? Has anyone
successfully used EBS?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/0d460d38-78a3-444b-9dd2-542f16015393%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/0d460d38-78a3-444b-9dd2-542f16015393%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b73a9c3e-2acd-461d-8d45-32fa4c900d59%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Thanks David. PIOPS will make this solution extremely expensive (in the
thousands of $) so I'll have to skip it.

I'll try to test EBS SSD on my own. Wondered if someone has already done
that and can share insights from their experience.

Thanks.

On Saturday, November 15, 2014 4:30:20 PM UTC+2, David Severski wrote:

With the relatively recent transition of default EBS from magnetic to
SSD-backed storage with burstable IO, I suggest you test out your workloads
and see what meets your required performance levels. I've successfully run
ES on slow magentic-EBS when I couldn't justify the cost of PIOPS+EBS
optimized instances but your needs may be (read: probably are) different
than mine.

David

On Saturday, November 15, 2014 12:57:18 AM UTC-8, David Pilato wrote:

If you use EBS, then ask for provisioned IO.

HTH

David

Le 15 nov. 2014 à 09:01, Tomer Levy tomer...@gmail.com a écrit :

Hello,

I'm trying to run Elasticsearch on EC2. After watching the webinar on
running Elasticsearch on EC2, It was mentioned that the only working
solution is using ephemeral storage since EBS is not fast enough.
However, the current EC2 instance M3.large or even m3.xlarge come with
very small disks that won't be enough (40GB). I want to ship 25GB a day and
retain data for 30 days.

How do you recommend that I go about setting it up? Has anyone
successfully used EBS?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/0d460d38-78a3-444b-9dd2-542f16015393%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/0d460d38-78a3-444b-9dd2-542f16015393%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/dd0eb2fe-8113-424d-8634-6cb1c033dc0b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Do you have a particular response time requirement for your ES cluster?
I'd suggest starting with the cheapest option available (magnetic EBS),
then moving up the ladder to the more expensive options until you're happy
with the speed/cost tradeoff. I've run ES clusters on magnetic EBS volumes
with no problem (e.g. for low-to-moderate volume logstash/kibana/marvel
clusters).

EBS is definitely slower than ephemeral (since it's just NAS), but I think
the webinar was maybe a bit quick to discount it. EBS didn't work for them
at their scale, but that doesn't mean it won't work for you!

Ross

On Sunday, 16 November 2014 04:40:09 UTC+11, Tomer Levy wrote:

Thanks David. PIOPS will make this solution extremely expensive (in the
thousands of $) so I'll have to skip it.

I'll try to test EBS SSD on my own. Wondered if someone has already done
that and can share insights from their experience.

Thanks.

On Saturday, November 15, 2014 4:30:20 PM UTC+2, David Severski wrote:

With the relatively recent transition of default EBS from magnetic to
SSD-backed storage with burstable IO, I suggest you test out your workloads
and see what meets your required performance levels. I've successfully run
ES on slow magentic-EBS when I couldn't justify the cost of PIOPS+EBS
optimized instances but your needs may be (read: probably are) different
than mine.

David

On Saturday, November 15, 2014 12:57:18 AM UTC-8, David Pilato wrote:

If you use EBS, then ask for provisioned IO.

HTH

David

Le 15 nov. 2014 à 09:01, Tomer Levy tomer...@gmail.com a écrit :

Hello,

I'm trying to run Elasticsearch on EC2. After watching the webinar on
running Elasticsearch on EC2, It was mentioned that the only working
solution is using ephemeral storage since EBS is not fast enough.
However, the current EC2 instance M3.large or even m3.xlarge come with
very small disks that won't be enough (40GB). I want to ship 25GB a day and
retain data for 30 days.

How do you recommend that I go about setting it up? Has anyone
successfully used EBS?

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/0d460d38-78a3-444b-9dd2-542f16015393%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/0d460d38-78a3-444b-9dd2-542f16015393%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7c367ba8-bcc1-4e8a-af4f-b39e8dad493c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

You can try using https://github.com/stec-inc/EnhanceIO which create block
device with magnetic disks and SSDs as cache.

On Saturday, November 15, 2014 10:01:20 AM UTC+2, Tomer Levy wrote:

Hello,

I'm trying to run Elasticsearch on EC2. After watching the webinar on
running Elasticsearch on EC2, It was mentioned that the only working
solution is using ephemeral storage since EBS is not fast enough.
However, the current EC2 instance M3.large or even m3.xlarge come with
very small disks that won't be enough (40GB). I want to ship 25GB a day and
retain data for 30 days.

How do you recommend that I go about setting it up? Has anyone
successfully used EBS?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/66202533-8419-470c-a560-b90379da636e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.