I'm trying to run Elasticsearch on EC2. After watching the webinar on
running Elasticsearch on EC2, It was mentioned that the only working
solution is using ephemeral storage since EBS is not fast enough.
However, the current EC2 instance M3.large or even m3.xlarge come with very
small disks that won't be enough (40GB). I want to ship 25GB a day and
retain data for 30 days.
How do you recommend that I go about setting it up? Has anyone successfully
used EBS?
I'm trying to run Elasticsearch on EC2. After watching the webinar on running Elasticsearch on EC2, It was mentioned that the only working solution is using ephemeral storage since EBS is not fast enough.
However, the current EC2 instance M3.large or even m3.xlarge come with very small disks that won't be enough (40GB). I want to ship 25GB a day and retain data for 30 days.
How do you recommend that I go about setting it up? Has anyone successfully used EBS?
With the relatively recent transition of default EBS from magnetic to
SSD-backed storage with burstable IO, I suggest you test out your workloads
and see what meets your required performance levels. I've successfully run
ES on slow magentic-EBS when I couldn't justify the cost of PIOPS+EBS
optimized instances but your needs may be (read: probably are) different
than mine.
David
On Saturday, November 15, 2014 12:57:18 AM UTC-8, David Pilato wrote:
If you use EBS, then ask for provisioned IO.
HTH
David
Le 15 nov. 2014 à 09:01, Tomer Levy <tomer...@gmail.com <javascript:>> a
écrit :
Hello,
I'm trying to run Elasticsearch on EC2. After watching the webinar on
running Elasticsearch on EC2, It was mentioned that the only working
solution is using ephemeral storage since EBS is not fast enough.
However, the current EC2 instance M3.large or even m3.xlarge come with
very small disks that won't be enough (40GB). I want to ship 25GB a day and
retain data for 30 days.
How do you recommend that I go about setting it up? Has anyone
successfully used EBS?
Thanks David. PIOPS will make this solution extremely expensive (in the
thousands of $) so I'll have to skip it.
I'll try to test EBS SSD on my own. Wondered if someone has already done
that and can share insights from their experience.
Thanks.
On Saturday, November 15, 2014 4:30:20 PM UTC+2, David Severski wrote:
With the relatively recent transition of default EBS from magnetic to
SSD-backed storage with burstable IO, I suggest you test out your workloads
and see what meets your required performance levels. I've successfully run
ES on slow magentic-EBS when I couldn't justify the cost of PIOPS+EBS
optimized instances but your needs may be (read: probably are) different
than mine.
David
On Saturday, November 15, 2014 12:57:18 AM UTC-8, David Pilato wrote:
I'm trying to run Elasticsearch on EC2. After watching the webinar on
running Elasticsearch on EC2, It was mentioned that the only working
solution is using ephemeral storage since EBS is not fast enough.
However, the current EC2 instance M3.large or even m3.xlarge come with
very small disks that won't be enough (40GB). I want to ship 25GB a day and
retain data for 30 days.
How do you recommend that I go about setting it up? Has anyone
successfully used EBS?
Do you have a particular response time requirement for your ES cluster?
I'd suggest starting with the cheapest option available (magnetic EBS),
then moving up the ladder to the more expensive options until you're happy
with the speed/cost tradeoff. I've run ES clusters on magnetic EBS volumes
with no problem (e.g. for low-to-moderate volume logstash/kibana/marvel
clusters).
EBS is definitely slower than ephemeral (since it's just NAS), but I think
the webinar was maybe a bit quick to discount it. EBS didn't work for them
at their scale, but that doesn't mean it won't work for you!
Ross
On Sunday, 16 November 2014 04:40:09 UTC+11, Tomer Levy wrote:
Thanks David. PIOPS will make this solution extremely expensive (in the
thousands of $) so I'll have to skip it.
I'll try to test EBS SSD on my own. Wondered if someone has already done
that and can share insights from their experience.
Thanks.
On Saturday, November 15, 2014 4:30:20 PM UTC+2, David Severski wrote:
With the relatively recent transition of default EBS from magnetic to
SSD-backed storage with burstable IO, I suggest you test out your workloads
and see what meets your required performance levels. I've successfully run
ES on slow magentic-EBS when I couldn't justify the cost of PIOPS+EBS
optimized instances but your needs may be (read: probably are) different
than mine.
David
On Saturday, November 15, 2014 12:57:18 AM UTC-8, David Pilato wrote:
I'm trying to run Elasticsearch on EC2. After watching the webinar on
running Elasticsearch on EC2, It was mentioned that the only working
solution is using ephemeral storage since EBS is not fast enough.
However, the current EC2 instance M3.large or even m3.xlarge come with
very small disks that won't be enough (40GB). I want to ship 25GB a day and
retain data for 30 days.
How do you recommend that I go about setting it up? Has anyone
successfully used EBS?
On Saturday, November 15, 2014 10:01:20 AM UTC+2, Tomer Levy wrote:
Hello,
I'm trying to run Elasticsearch on EC2. After watching the webinar on
running Elasticsearch on EC2, It was mentioned that the only working
solution is using ephemeral storage since EBS is not fast enough.
However, the current EC2 instance M3.large or even m3.xlarge come with
very small disks that won't be enough (40GB). I want to ship 25GB a day and
retain data for 30 days.
How do you recommend that I go about setting it up? Has anyone
successfully used EBS?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.