Moving to production considerations - applications with their clusters' infrastructure

Elastic Stack Elasticsearch
1

Hi everyone,

I am considering moving one or several elasticsearch clusters to production.
Although Elasticsearch's documentation and community is great, I am
strongly startled not to find any complete use-case story stretching from
application(s) needs and data considerations to hardware ones.
Indeed, I understand why "what/how much hardware / configuration /
sharding" questions are systematically replied with both "it depends"
followed by "test".
But then, what about a few complete descriptions, out of so many
elasticsearch users, from data use case to cluster's internals, along with
a few performance and nodes stats?

So here are questions, before moving to production :

Are there any complete use cases around? Could you share some? By
complete I mean including at least some of the following :

  1. Application needs and scope
  2. Indexing Data indications : data volume, documents mapping,
    documents / indexes volume
  3. Searching Data indications : different applications, queries, use
    of facets - filters - aggregations, concurrent indexing
  4. Cluster Hardware : machines' hardware (RAM, Disks/SSD -
    DAS-JBOD/SAN/NAS), JVM heap / OS Cache, nb of machines, back office network
  5. Cluster Configuration : one or several indexes, sharding,
    replication, master nodes, data nodes, use of over-sharding at start-up,
    use of re-indexing
  6. *Benchmaks *: queries response times, QPS, with or without concurrent
    indexing, memory heap sweet spot, nodes stats

For those interested, here are the (not complete) best-among-very-few
exemples I've stumbled upon so far :

https://speakerdeck.com/bhaskarvk/scaling-elasticsearch-washington-dc-meetup
With JBOD / SAN storage discussion in "To Raid or not to Raid":

https://groups.google.com/forum/?fromgroups#!searchin/elasticsearch/hardware/elasticsearch/HSj2fZGdU1Y/4mFCBTCb-JcJ

  • Usual heap considerations in a real case :

https://codeascraft.com/2014/12/04/juggling-multiple-elasticsearch-instances-on-a-single-host/

Do not forget Elasticsearch awesome docs for moving to production
considerations :

http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/administration.html

http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/deploy.html

http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/hardware.html

http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/heap-sizing.html

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/927f60b1-8ae2-463e-b725-5f4f993905d9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Hi,

Firstly, thanks for the kind words about the performance study: we're
hoping to revisit this soon after the feedback we've had on better tuning
for each engine. I agree there's a paucity of studies, but in a month or
two we should have one from a project we're working on to build an index of
product data - we hope to present this at the London Meetup group but will
publish the slides afterwards. Sorry this is in the future!

Cheers

Charlie

On 12 February 2015 at 14:27, rondelvictor@gmail.com wrote:

Hi everyone,

I am considering moving one or several elasticsearch clusters to
production.
Although Elasticsearch's documentation and community is great, I am
strongly startled not to find any complete use-case story stretching
from application(s) needs and data considerations to hardware ones.
Indeed, I understand why "what/how much hardware / configuration /
sharding" questions are systematically replied with both "it depends"
followed by "test".
But then, what about a few complete descriptions, out of so many
elasticsearch users, from data use case to cluster's internals, along with
a few performance and nodes stats?

So here are questions, before moving to production :

Are there any complete use cases around? Could you share some? By
complete I mean including at least some of the following :

  1. Application needs and scope
  2. Indexing Data indications : data volume, documents mapping,
    documents / indexes volume
  3. Searching Data indications : different applications, queries, use
    of facets - filters - aggregations, concurrent indexing
  4. Cluster Hardware : machines' hardware (RAM, Disks/SSD -
    DAS-JBOD/SAN/NAS), JVM heap / OS Cache, nb of machines, back office network
  5. Cluster Configuration : one or several indexes, sharding,
    replication, master nodes, data nodes, use of over-sharding at start-up,
    use of re-indexing
  6. *Benchmaks *: queries response times, QPS, with or without
    concurrent indexing, memory heap sweet spot, nodes stats

For those interested, here are the (not complete) best-among-very-few
exemples I've stumbled upon so far :

https://speakerdeck.com/bhaskarvk/scaling-elasticsearch-washington-dc-meetup
With JBOD / SAN storage discussion in "To Raid or not to Raid":

Redirecting to Google Groups

  • Usual heap considerations in a real case :

https://codeascraft.com/2014/12/04/juggling-multiple-elasticsearch-instances-on-a-single-host/

Do not forget Elasticsearch awesome docs for moving to production
considerations :

Elasticsearch Platform — Find real-time answers at scale | Elastic

Elasticsearch Platform — Find real-time answers at scale | Elastic

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/927f60b1-8ae2-463e-b725-5f4f993905d9%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/927f60b1-8ae2-463e-b725-5f4f993905d9%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGe-ML%2B1EXPXxF6B1yr_ubcOFTgAPJQFUwJjY2b-Ffigv1zr%3Dg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

Hi Charlie,

That's excellent news! Thank you for your slideshare and related github!

Regards,

Victor

Le vendredi 13 février 2015 16:07:11 UTC+1, Charlie Hull a écrit :

Hi,

Firstly, thanks for the kind words about the performance study: we're
hoping to revisit this soon after the feedback we've had on better tuning
for each engine. I agree there's a paucity of studies, but in a month or
two we should have one from a project we're working on to build an index of
product data - we hope to present this at the London Meetup group but will
publish the slides afterwards. Sorry this is in the future!

Cheers

Charlie

On 12 February 2015 at 14:27, <rondel...@gmail.com <javascript:>> wrote:

Hi everyone,

I am considering moving one or several elasticsearch clusters to
production.
Although Elasticsearch's documentation and community is great, I am
strongly startled not to find any complete use-case story stretching
from application(s) needs and data considerations to hardware ones.
Indeed, I understand why "what/how much hardware / configuration /
sharding" questions are systematically replied with both "it depends"
followed by "test".
But then, what about a few complete descriptions, out of so many
elasticsearch users, from data use case to cluster's internals, along with
a few performance and nodes stats?

So here are questions, before moving to production :

Are there any complete use cases around? Could you share some? By
complete I mean including at least some of the following :

  1. Application needs and scope
  2. Indexing Data indications : data volume, documents mapping,
    documents / indexes volume
  3. Searching Data indications : different applications, queries,
    use of facets - filters - aggregations, concurrent indexing
  4. Cluster Hardware : machines' hardware (RAM, Disks/SSD -
    DAS-JBOD/SAN/NAS), JVM heap / OS Cache, nb of machines, back office network
  5. Cluster Configuration : one or several indexes, sharding,
    replication, master nodes, data nodes, use of over-sharding at start-up,
    use of re-indexing
  6. *Benchmaks *: queries response times, QPS, with or without
    concurrent indexing, memory heap sweet spot, nodes stats

For those interested, here are the (not complete) best-among-very-few
exemples I've stumbled upon so far :

https://speakerdeck.com/bhaskarvk/scaling-elasticsearch-washington-dc-meetup
With JBOD / SAN storage discussion in "To Raid or not to Raid":

Redirecting to Google Groups

  • Usual heap considerations in a real case :

https://codeascraft.com/2014/12/04/juggling-multiple-elasticsearch-instances-on-a-single-host/

Do not forget Elasticsearch awesome docs for moving to production
considerations :

Elasticsearch Platform — Find real-time answers at scale | Elastic

Elasticsearch Platform — Find real-time answers at scale | Elastic

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/927f60b1-8ae2-463e-b725-5f4f993905d9%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/927f60b1-8ae2-463e-b725-5f4f993905d9%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/34acb9f4-a622-4ed6-b877-d605f0c6382b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.