Yes, of course Elasticsearch is using off-heap memory. All the Lucene index
I/O is using direct buffers in native OS memory.
Errors in allocating direct buffers will result in Java errors. You mention
Linux memory errors but unfortunately you do not quote it, so I have to
guess.
You should have enabled memory mapped files by index store mmapfs (default
on RHEL) so all files that are read by ES are mapped into virtual address
space of the OS VM management.
And also bootstrap.mlockall = true, so you also need to set memlock to
unlimited in /etc/security/limits.conf, because RHEL/Centos memlockable
memory is limited to 25% of RAM by default. In that case, Java should throw
an IOException "Map failed".
Note, because of the memory page lock support of the host OS, you should
also check what kind of virtualization you have enabled for the guest, it
should be HW (full) virtualization, not paravirtualization.
If you still encounter issues from Linux OS errors it is most probably
because of VMware limitations, so you should disable the bootstrap.mlockall
setting.
As a side note, the recommended heap size is 50% of the RAM that is
available to the ES process. If you run a VM, you should assign at most 50%
of the configured guest OS memory to ES.
Jörg
On Tue, May 6, 2014 at 10:35 PM, Edward Sargisson ejsarge@gmail.com wrote:
Hi all,
We have a problem where our es nodes will fail with an out of memory error
from Linux (note, not Java). Our es processes are configured with a fixed
amount of heap (60% of total RAM - just as in in the elasticsearch chef
cookbook).So, something is consuming all of the memory available to Linux.
Is there any other memory that ES can use? Does it lock OS cache or buffer
memory so that it can't be released? If it opens lots of files does it use
up too much RAM? Is it doing off-heap allocation? (I'm pretty sure the
answer is no to the last).We're struggling to find the exact memory resource being used up.
For the record. this is ES 1.1.0 on CentOS 6.4 running in VMWare.
Thanks!
Edward--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/ab6421e3-89a1-409f-b89b-f09ca5bc9551%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/ab6421e3-89a1-409f-b89b-f09ca5bc9551%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoF%3Dw7iyBUfKxYQm45yG6Zh6a5Rg7SipDKNmdPA3MijYGw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.