Creating and storing ES indices on S3


(Lavesh Gupta) #1

Hi Everyone,

I went through the group and came across various replies by David Pilato,
Kimchy and other people which stated to use EBS volumes or local disks for
storing ES indices and S3 for periodic snapshots. Wanted to know the reason
why using S3 will be a bad choice for creating and storing indices on S3
(even if I mount S3 to EC2 instance).

My main points for storing and creating indices on S3 are:

  1. I want to avoid cost of using both EBS and S3. (EBS for storing indices
    data and S3 for snapshots). If I only use S3 then due to replication
    feature I do not have to worry about my data loss.
  2. Using S3 I will not have to worry about failure of nodes as all data is
    in centralized placed being replicated.
  3. I can use AWS auto scaling feature to much advantage as I can spawn a
    new EC2 machine when demand rises and then all the indexing and querying
    can be done through a central data source on S3.

If somebody can suggest how I can achieve creating and storing of indices
on S3 it will be a great help.

Thanking You

--

This message contains confidential information and is intended only for the
individual to whom it is addressed. If you are not the intended recipient,
you should not disseminate, distribute or copy this e-mail. Please notify
the sender immediately by e-mail if you have received this e-mail by
mistake and permanently delete this e-mail from your system. E-mail
transmission cannot be guaranteed to be secure or error-free as information
could be intercepted, corrupted, lost, destroyed, late or incomplete, or
could contain viruses. The sender therefore does not accept liability for
any errors or omissions in the contents of this message, which arise as a
result of e-mail transmission. If verification is required, please request
a hard-copy version from the sender. Druva, www.druva.com

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/77357d50-d938-4c4b-9a4b-98001e793798%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Mark Walkom) #2

It might be possible, but not really recommended as I'd imagine it'd be
pretty slow, which would impact your performance of ES.

On 4 May 2015 at 23:56, Lavesh Gupta lavesh.gupta@druva.com wrote:

Hi Everyone,

I went through the group and came across various replies by David Pilato,
Kimchy and other people which stated to use EBS volumes or local disks for
storing ES indices and S3 for periodic snapshots. Wanted to know the reason
why using S3 will be a bad choice for creating and storing indices on S3
(even if I mount S3 to EC2 instance).

My main points for storing and creating indices on S3 are:

  1. I want to avoid cost of using both EBS and S3. (EBS for storing indices
    data and S3 for snapshots). If I only use S3 then due to replication
    feature I do not have to worry about my data loss.
  2. Using S3 I will not have to worry about failure of nodes as all data is
    in centralized placed being replicated.
  3. I can use AWS auto scaling feature to much advantage as I can spawn a
    new EC2 machine when demand rises and then all the indexing and querying
    can be done through a central data source on S3.

If somebody can suggest how I can achieve creating and storing of indices
on S3 it will be a great help.

Thanking You

This message contains confidential information and is intended only for
the individual to whom it is addressed. If you are not the intended
recipient, you should not disseminate, distribute or copy this e-mail.
Please notify the sender immediately by e-mail if you have received this
e-mail by mistake and permanently delete this e-mail from your system.
E-mail transmission cannot be guaranteed to be secure or error-free as
information could be intercepted, corrupted, lost, destroyed, late or
incomplete, or could contain viruses. The sender therefore does not accept
liability for any errors or omissions in the contents of this message,
which arise as a result of e-mail transmission. If verification is
required, please request a hard-copy version from the sender. Druva,
www.druva.com

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/77357d50-d938-4c4b-9a4b-98001e793798%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/77357d50-d938-4c4b-9a4b-98001e793798%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8UV_%3DEBTLFdvDev4%2BMpVaOUwc7Yw6wBKve3TdO1CnpzQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(system) #3