Trouble understanding mlockall behaviour

Elastic Stack Elasticsearch
1

Hi,

Following the advice on different blog post on production setup for
Elasticsearch, I'm using -Xms256m -Xmx256 and set bootstrap.mlockall to
True.
My guess was that the after restart, my Elasticsearch process would use
about 256m of ram, and that it's memory usage would remain quasi-constant
during it's lifetime.
But after the restart, my Elasticsearch process in using about 556m of ram.

So, can someone please explain me the behaviour of mlockall ?

THx in advance.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1c761cf3-98d6-4010-bd46-50849c48b760%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

This is what it does - http://linux.die.net/man/2/mlockall
Basically it prevents the memory being moved to swap.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 30 January 2014 00:01, briche arnaud briche.arnaud@gmail.com wrote:

Hi,

Following the advice on different blog post on production setup for
Elasticsearch, I'm using -Xms256m -Xmx256 and set bootstrap.mlockall to
True.
My guess was that the after restart, my Elasticsearch process would use
about 256m of ram, and that it's memory usage would remain quasi-constant
during it's lifetime.
But after the restart, my Elasticsearch process in using about 556m of ram.

So, can someone please explain me the behaviour of mlockall ?

THx in advance.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1c761cf3-98d6-4010-bd46-50849c48b760%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624Z31jzjtLaZVQ_GwSqf-x6ivDEH-2ObC_zPMwrzgd7ckA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Also mlockall may not be supported depending on your OS.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6fa34ed9-305e-4e94-8c3d-8f7683dae71a%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

The JVM uses reserved memory, buffers, and stacks beside the heap, most can
not be controlled by user settings.
To reduce memory as much as possible for a Java process, do not use
mlockall tricks, use 32bit JVM and reduce maximum allowed stack size with
-Xss (this will affect performance of course)

You will also need many tweaks of Elasticsearch default settings to behave
under shrinked memory, for index loading, field caches, etc.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFkHRUsTt-0s7vV_%2BCNL%2BdUW-erVS4JF_OiL%2B4h-X4mXg%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

Ok, I now understand mlockall. But I'm not asking this question in the
context of a memory contrived node; my production nodes have 32G ram; I
used this setting in order to prevent OOM of prodution nodes.

I may misunderstood the informations a gathered from the web, but I deduced
that with the following config params :

bootstrap.mlockall: true
index.fielddata.cache: node
indices.fielddata.cache.size: 40%

I would have better control of the memory usage of ES nodes.
I still don't understand though, why a node started with -Xms256m -Xmx256 +
mlockall: false only take 80M ram while a node started with -Xms256m
-Xmx256 + mlockall: true take 600M.

On Thu, Jan 30, 2014 at 1:34 AM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

The JVM uses reserved memory, buffers, and stacks beside the heap, most
can not be controlled by user settings.
To reduce memory as much as possible for a Java process, do not use
mlockall tricks, use 32bit JVM and reduce maximum allowed stack size with
-Xss (this will affect performance of course)

You will also need many tweaks of Elasticsearch default settings to behave
under shrinked memory, for index loading, field caches, etc.

Jörg

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/DijPr5PNTYM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFkHRUsTt-0s7vV_%2BCNL%2BdUW-erVS4JF_OiL%2B4h-X4mXg%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
-- Arnaud Briche
Développeur web / Préstataire de service informatique

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CADP2VkAD14qs0_dV7%2BgtaSOrBR3o0rJBWcc-xsyrHH%3DAHXp8cA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

6

If you have 32G RAM, just configure ES with

ES_HEAP_SIZE=16g

If you want to mess with JVM options, set them to -Xmx16g -Xms16g (look
into bin/elasticsearch.in.sh assuming you are on Linux)

To answer your question, you have passed wrong args in Xmx to the JVM and
the JVM silently resets the heap to the initial heap size, which you can
find out with the command

java -XX:+PrintFlagsFinal -version | grep HeapSize

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoENgdThwdEiktDXLPyFECcsDLBK7kOkAnj-OqV89kbiDg%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.