RAM / memory sizing

Elastic Stack Elasticsearch
1

Hi together,

I need to size memory of a elasticsearch cluster. I expect an total index
size of 5 terra byte.

For example: If I would use 5 server with 1 terra byte harddrives,

what should each server have for RAM sizes to get great read performance?

Is there a rule of thumb for sizing server RAM based on index size?

Best regards
Steffen

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9581f499-bef6-4437-86f4-8620d517fb98%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

I assume you mean query performance when you mention "great read
performance"

It depends on the queries. Several factors are to consider:

  • data characteristics (number of fields, number of terms, number of docs,
    immutable/mutable data, change rate,...)
  • query types/complexities (simple term queries, boolean queries, filters,
    caching, aggregations,...)
  • query load (how many searches must be performed in a given time interval)

Regarding an index size as large as TB, I think it would be best to put a
logical ordering on it and split the index into many indices, e.g. into
timeseries indices or into user id based indices, and combine indices with
an index alias to a logical unit. Then sliding window techniques can be
used to focus the search only on indices which are relevant for expected
responses. This allows to control hot spots in your data, and it can save
enormous amounts of resources for replica shards.

Jörg

On Fri, Jul 18, 2014 at 6:51 PM, Steffen Otto <steffen.otto22@googlemail.com

wrote:

Hi together,

I need to size memory of a elasticsearch cluster. I expect an total index
size of 5 terra byte.

For example: If I would use 5 server with 1 terra byte harddrives,

what should each server have for RAM sizes to get great read performance?

Is there a rule of thumb for sizing server RAM based on index size?

Best regards
Steffen

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/9581f499-bef6-4437-86f4-8620d517fb98%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/9581f499-bef6-4437-86f4-8620d517fb98%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFKJz-fcvGmxx7%3DsUFaB27BREzWsydsCwQJfOiNSvG-gQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

Hi Jörg, thank you for your answer but you did not answer my question.
Is there a general rule of thumb for RAM sizing a machine when I expect an fixed index size for great query performance?
And also is there a upper size where more RAM does not make more sense?
Best regards
Steffen

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8c565999-1dc2-4591-a79d-4dc815e50352%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

No, there is no general rule of thumb, since there is no direct correlation
between a fixed index size and the demand for RAM.

There is no upper size limit for RAM imposed by Elasticsearch. JVM
implementations imposes some restrictions, for heaps > 4g, garbage
collecting the heap objects becomes challenging. This is true for all Java
server applications.

Jörg

On Fri, Jul 18, 2014 at 7:33 PM, Steffen Otto <steffen.otto22@googlemail.com

wrote:

Hi Jörg, thank you for your answer but you did not answer my question.
Is there a general rule of thumb for RAM sizing a machine when I expect an
fixed index size for great query performance?
And also is there a upper size where more RAM does not make more sense?
Best regards
Steffen

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8c565999-1dc2-4591-a79d-4dc815e50352%40googlegroups.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoF3ysHg%2BmOU-jCMharPBx_b3PRJucMcbGp425wAbTurwQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.