Larger documents maybe 1-2k a piece. We do aggregations/facets with some search.
What heap should we have on a node with 60gb of memory?. Right now we get an occasional out of memory and long GCs.
—
Thanks
On Sun, Dec 21, 2014 at 3:47 PM, Mark Walkom markwalkom@gmail.com wrote:
That's way too many shards, you only really need 1 shard per node, unless
you're expecting to dramatically increase your node count in the near
future.
More info on what data it is would be useful; are the docs large, numerous,
are you using relationships as I mentioned?
What about your querying? Lots of aggregations/facets?
And as I mentioned, you lose performance by having a 32GB heap, I'd
strongly recommonend you reduce it.
On 21 December 2014 at 20:04, Jeff Rick jeffrick@gmail.com wrote:So each node has 60gb of memory and heap is set to 32gb.
This is Medical data, we do lots of aggregations but no parent child
relationships. We currently have 36 shards for the index.—
ThanksOn Sun, Dec 21, 2014 at 2:53 PM, Mark Walkom markwalkom@gmail.com wrote:
You should reduce your heap to half your system RAM, ie 30GB, so it's
not more than 31GB. Above that your java pointers aren't compressed and you
get less efficient heap use.What sort of data is it, how many shards in your index, are your queries
heavy (ie lots of aggs), are you using parent+child relationships?On 21 December 2014 at 16:10, Jeff Rick jeffrick@gmail.com wrote:
Hi,
So we have a small development cluster that we are doing some sizing
tests with. The make-up of the cluster as is follows:4 Data Nodes - 60GB of memory and 108 cores (aka c3.8xlarge) - 32GB per
node for heap
1 Query / Master Node - 61GB memory and 26 cores (aka r3.2xlarge) -
32GB per node for heapWe have around 280GB of data and 250M documents in a single index. This
index is a readonly index, we do no indexing. My question is what is using
the heap?I am showing 82GB of heap used of 155GB total. The Field Data Cache is
using 16GB and the Filter Cache is using 12 GB with another 1GB being used
by Lucene. Query Cache on the data nodes is next to nothing and on the
query node it is a less than a GB. So what could be using the remaining
50GB?--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/d82d323e-27e2-4a44-8c56-c5864b0cf7f9%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/d82d323e-27e2-4a44-8c56-c5864b0cf7f9%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/6HriDQmIikI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_6_iJ8dRmdrvx1gPDTJDDzDPb0xPsEzKCGs1psmB5-hg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_6_iJ8dRmdrvx1gPDTJDDzDPb0xPsEzKCGs1psmB5-hg%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1419192277058.7d049799%40Nodemailer
https://groups.google.com/d/msgid/elasticsearch/1419192277058.7d049799%40Nodemailer?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/6HriDQmIikI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9pX8MR3REZMZ5dvw%2BPC%2B2-Jxes6_41Pom1MwS0VgZtVw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1419197214304.1e105e4b%40Nodemailer.
For more options, visit https://groups.google.com/d/optout.