Hello Anand,
Yes, your search performance will probably depend a lot on the RAM you have
available. Unfortunately, at least I can't give you a RAM/Index_size ratio,
not even a rough one, because that would depend heavily on:
- how often your indices are hit by searches
- how complex the searches are
- how your documents look like (how many fields, how they're analyzed...)
Regarding memory itself, there are two things:
- the memory ES uses. This is where it's important to give the JVM enough
room for Elasticsearch's needs, otherwise the Garbage Collector will eat
your CPU "recycling your RAM" all the time. And, especially if you searches
are heavy, you want CPU to take care of searches, instead of GC
- the memory OS caches use. Because you don't want ES to hit your disk
continuously to get indexed data, the OS caches it. The more of your ES
index can fit in your "free" RAM, the faster your searches will be. If your
indices are waaay larger than your RAM, and I/O seems to be a bottleneck,
SSDs may be a good option
Back to your RAM question, here's what I would do:
- take the data set you want to search on and put it in a test cluster. Or
a portion of that data, with a portion of what might be your production
cluster
- allocate 50% of your RAM to ES. That's usually a good start
- run some tests that load your test cluster as you expect a production
cluster to be loaded
- monitor the cluster with some monitoring tool like our
SPMhttp://sematext.com/spm/elasticsearch-performance-monitoring/
- check out how much memory your JVM uses. It will go up and down, but
you'll get a rough idea. For example, let's say you have 32GB of RAM in
total, and you've allocated 16 to ES initially. And you notice that your
average memory usage for the JVM is about 5GB. In this case, you'll
probably be better off lowering ES_HEAP_SIZE to 10 or maybe 8 GB, and leave
more room for OS caches.
On the other hand, if you see the GC running often, and your memory usage
going up and down close to your 16GB limit, it's a sign that ES needs more
RAM.
You can also dig deeper into what eats your RAM. If you use filters a lot,
ES will rely on filter caches. If you sort or facet a lot, ES will rely on
field caches. You can look at how these caches are used in SPM and tweak
them in the ES configuration.
Best regards,
Radu
http://sematext.com/ -- Elasticsearch -- Solr -- Lucene
On Thu, Jul 18, 2013 at 5:29 PM, Anand Nalya anand.nalya@gmail.com wrote:
Hi,
I understand that search performance for ES is directly affected by the
available RAM. but I still want to know what all steps can be taken to
decrease the amount of RAM requirements.
Generally what is the ration of RAM/Index Size has been observed in
production.
Regards,
Anand
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.