MIN/MAX memory allocation and mlockall

Elastic Stack Elasticsearch
1

We're running Dell R720xds with 512G RAM, ES 0.90.0, Java 1.7.0_25 and
Ubuntu 12.04 and we're wanting to allocate ES 256G of RAM (ie 50% of
physical).

However during startup we were noticing that ES would continually core
dump. We ended up having to set;

ES_MIN_MEM=64g
ES_MAX_MEM=256g

Where before we were using;

ES_HEAP_SIZE=256g

Just to get ES to start and connect to the existing cluster.

This does appear to be a java bug occurring when trying to allocate the
complete xms. Our main concern is if there be any issues if we continue to
set "bootstrap.mlockall: true", as this setup doesn't align with the
official documentation?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

No lunch is for free.

I would be nice to get the core dump for analysis, so it is not possible to
help.

So I have only general hints, where YMMV.

First, check if your Linux gives you enough resources for mmap() with
ulimit -l (for the mlockall call by JNA).

Second, please note, that you must also reserve swap for the size of your
memory space you demand with Xms (which in the end is something like a
sequence of simple malloc()'s on fragments of contiguous space). So, check
if your system offers 256G contiguous space, and if the swap area is at
least 256G (plus some GB for non-heap memory). When you start the JVM
process with Xms, Linux must be able to move other pages out of the way to
swap, if you already had pages allocated which are swappable. If you don't,
Linux let the process crash. By default, Linux reserves 25% of total RAM
for it (and this matches your 64G setting).

Third, personally I'm curious how you analyzed that only such a large heap
memory is feasible for JVM apps, like ES. Have you run other apps with such
a setting succesfully? I must admit I doubt it. For example, managing
caches in such a large heap is a nightmare, and I predict it will slow down
your ES significantly. Also, GC will get slow, compared to less aggressive
heap setting, like 8GB or 16GB.

Jörg

On Mon, Jul 29, 2013 at 7:36 AM, Mark Walkom markw@campaignmonitor.comwrote:

We're running Dell R720xds with 512G RAM, ES 0.90.0, Java 1.7.0_25 and
Ubuntu 12.04 and we're wanting to allocate ES 256G of RAM (ie 50% of
physical).

However during startup we were noticing that ES would continually core
dump. We ended up having to set;

ES_MIN_MEM=64g
ES_MAX_MEM=256g

Where before we were using;

ES_HEAP_SIZE=256g

Just to get ES to start and connect to the existing cluster.

This does appear to be a java bug occurring when trying to allocate the
complete xms. Our main concern is if there be any issues if we continue to
set "bootstrap.mlockall: true", as this setup doesn't align with the
official documentation?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

ulimit is pretty high, 512000. The total filesystem is 10TB and most of
that is empty, so a contiguous 256GB shouldn't be a problem.

We have 4G of swap as we were of the impression that given the large amount
of RAM it wouldn't need a lot of swap, just enough to allow the system to
operate and occasionally use. This was based on readings such as

.
However based on your comments I am guessing that setting the xms to 256G
with such a small swap is probably why it wouldn't initially start.

These are new machines and we are experimenting with heap sizes etc, we're
happy to get comments on how feasible it is to run with such a large heap
size and if there are other methods we could use to allow ES efficient
access to the RAM.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 29 July 2013 17:13, joergprante@gmail.com joergprante@gmail.com wrote:

No lunch is for free.

I would be nice to get the core dump for analysis, so it is not possible
to help.

So I have only general hints, where YMMV.

First, check if your Linux gives you enough resources for mmap() with
ulimit -l (for the mlockall call by JNA).

Second, please note, that you must also reserve swap for the size of your
memory space you demand with Xms (which in the end is something like a
sequence of simple malloc()'s on fragments of contiguous space). So, check
if your system offers 256G contiguous space, and if the swap area is at
least 256G (plus some GB for non-heap memory). When you start the JVM
process with Xms, Linux must be able to move other pages out of the way to
swap, if you already had pages allocated which are swappable. If you don't,
Linux let the process crash. By default, Linux reserves 25% of total RAM
for it (and this matches your 64G setting).

Third, personally I'm curious how you analyzed that only such a large heap
memory is feasible for JVM apps, like ES. Have you run other apps with such
a setting succesfully? I must admit I doubt it. For example, managing
caches in such a large heap is a nightmare, and I predict it will slow down
your ES significantly. Also, GC will get slow, compared to less aggressive
heap setting, like 8GB or 16GB.

Jörg

On Mon, Jul 29, 2013 at 7:36 AM, Mark Walkom markw@campaignmonitor.comwrote:

We're running Dell R720xds with 512G RAM, ES 0.90.0, Java 1.7.0_25 and
Ubuntu 12.04 and we're wanting to allocate ES 256G of RAM (ie 50% of
physical).

However during startup we were noticing that ES would continually core
dump. We ended up having to set;

ES_MIN_MEM=64g
ES_MAX_MEM=256g

Where before we were using;

ES_HEAP_SIZE=256g

Just to get ES to start and connect to the existing cluster.

This does appear to be a java bug occurring when trying to allocate the
complete xms. Our main concern is if there be any issues if we continue to
set "bootstrap.mlockall: true", as this setup doesn't align with the
official documentation?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

ulimit -l value of 512000 kb (=512M) is by far not enough for using
mlockall option. Set it to unlimited.

The filesystem does not matter at all for contiguous RAM. The fragmentation
of the RAM is managed by the kernel. The easiest method to regain
contiguous RAM is reboot - but I assume you use only ES on the machine so
there should be no problem.

Jörg

On Mon, Jul 29, 2013 at 10:46 AM, Mark Walkom markw@campaignmonitor.comwrote:

ulimit is pretty high, 512000. The total filesystem is 10TB and most of
that is empty, so a contiguous 256GB shouldn't be a problem.

We have 4G of swap as we were of the impression that given the large
amount of RAM it wouldn't need a lot of swap, just enough to allow the
system to operate and occasionally use. This was based on readings such as
9.15.5. Recommended Partitioning Scheme Red Hat Enterprise Linux 6 | Red Hat Customer Portal
.
However based on your comments I am guessing that setting the xms to 256G
with such a small swap is probably why it wouldn't initially start.

These are new machines and we are experimenting with heap sizes etc, we're
happy to get comments on how feasible it is to run with such a large heap
size and if there are other methods we could use to allow ES efficient
access to the RAM.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 29 July 2013 17:13, joergprante@gmail.com joergprante@gmail.comwrote:

No lunch is for free.

I would be nice to get the core dump for analysis, so it is not possible
to help.

So I have only general hints, where YMMV.

First, check if your Linux gives you enough resources for mmap() with
ulimit -l (for the mlockall call by JNA).

Second, please note, that you must also reserve swap for the size of your
memory space you demand with Xms (which in the end is something like a
sequence of simple malloc()'s on fragments of contiguous space). So, check
if your system offers 256G contiguous space, and if the swap area is at
least 256G (plus some GB for non-heap memory). When you start the JVM
process with Xms, Linux must be able to move other pages out of the way to
swap, if you already had pages allocated which are swappable. If you don't,
Linux let the process crash. By default, Linux reserves 25% of total RAM
for it (and this matches your 64G setting).

Third, personally I'm curious how you analyzed that only such a large
heap memory is feasible for JVM apps, like ES. Have you run other apps with
such a setting succesfully? I must admit I doubt it. For example, managing
caches in such a large heap is a nightmare, and I predict it will slow down
your ES significantly. Also, GC will get slow, compared to less aggressive
heap setting, like 8GB or 16GB.

Jörg

On Mon, Jul 29, 2013 at 7:36 AM, Mark Walkom markw@campaignmonitor.comwrote:

We're running Dell R720xds with 512G RAM, ES 0.90.0, Java 1.7.0_25 and
Ubuntu 12.04 and we're wanting to allocate ES 256G of RAM (ie 50% of
physical).

However during startup we were noticing that ES would continually core
dump. We ended up having to set;

ES_MIN_MEM=64g
ES_MAX_MEM=256g

Where before we were using;

ES_HEAP_SIZE=256g

Just to get ES to start and connect to the existing cluster.

This does appear to be a java bug occurring when trying to allocate the
complete xms. Our main concern is if there be any issues if we continue to
set "bootstrap.mlockall: true", as this setup doesn't align with the
official documentation?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.