# Memory confusion in Ubuntu 16.04

**URL:** <https://discuss.elastic.co/t/memory-confusion-in-ubuntu-16-04/50961>\
**Category:** Elasticsearch\
**Created:** [May 25, 2016, 4:14pm UTC](https://discuss.elastic.co/t/memory-confusion-in-ubuntu-16-04/50961 "2016-05-25T16:14:07Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![maziyar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/maziyar/32/18423_2.png) [@maziyar](https://discuss.elastic.co/u/maziyar)\
**Post date:** [May 25, 2016, 4:14pm UTC](https://discuss.elastic.co/t/memory-confusion-in-ubuntu-16-04/50961/1 "2016-05-25T16:14:07Z")

</div>

Hello,

I am upgrading my Elasticsearch cluster with 4 nodes from 14.04 (ES 2.3.2) to Ubuntu 16.04 (ES 2.3.3). I upgraded the ES first to the latest 2.3.3 on Ubuntu 14.04 to see what happens and everything is green and functional. So I started to install a fresh Ubuntu 16.04 and Elasticsearch with my configs.

Now I am seeing these warnings. Also it fails to lock the memory. I have swap off (no partition for swap at all) and 60GB of memory on each machine and all it shows in top is less than 2GB:

```
[2016-05-25 17:35:37,125][WARN][bootstrap] Unable to lock JVM Memory: error=12,reason=Cannot allocate memory
[2016-05-25 17:35:37,126][WARN][bootstrap] This can result in part of the JVM being swapped out.
[2016-05-25 17:35:37,126][WARN][bootstrap] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
[2016-05-25 17:35:37,126][WARN][bootstrap] These can be adjusted by modifying /etc/security/limits.conf, for example:
        # allow user 'elasticsearch' mlockall
        elasticsearch soft memlock unlimited
        elasticsearch hard memlock unlimited
[2016-05-25 17:35:37,126][WARN][bootstrap] If you are logged in interactively, you will have to re-login for the new limits to take effect.
[2016-05-25 17:35:37,582][INFO][node] [cluster-es-1] version[2.3.3], pid[940], build[218bdf1/2016-05-17T15:40:04Z]
[2016-05-25 17:35:37,583][INFO][node] [cluster-es-1] initializing ...
[2016-05-25 17:35:38,755][INFO][plugins] [cluster-es-1] modules [reindex, lang-expression, lang-groovy], plugins [license, marvel-agent, kopf, graph], sites [kopf]
[2016-05-25 17:35:38,836][INFO][env] [cluster-es-1] using [1] data paths, mounts [[/data (/dev/vdb1)]], net usable_space [1.8tb], net total_space [1.9tb], spins? [possibly], types [ext4]
[2016-05-25 17:35:38,836][INFO][env] [cluster-es-1] heap size [19.9gb], compressed ordinary object pointers [true]
[2016-05-25 17:35:38,837][WARN][env] [cluster-es-1] max file descriptors [65535] for elasticsearch process likely too low, consider increasing to at least [65536]
[2016-05-25 17:35:42,242][INFO][node] [cluster-es-1] initialized
[2016-05-25 17:35:42,243][INFO][node] [cluster-es-1] starting ...

```

What am I missing?

/etc/default/elasticsearch

```
ES_HEAP_SIZE=20g
MAX_OPEN_FILES=65536
MAX_LOCKED_MEMORY=unlimited

```

/etc/elasticsearch/elasticsearch.yml

`bootstrap.mlockall: true`

/etc/security/limits.conf

```
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited
* soft nofile 999999
* hard nofile 999999
* soft memlock unlimited
* hard memlock unlimited
root soft nofile 999999
root hard nofile 999999
root soft memlock unlimited
root hard memlock unlimited

```

ulimit -a

```
file size (blocks, -f) unlimited
pending signals (-i) 241446
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 999999
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

```

Node on Ubuntu 14.04:

```
   "jvm": {
    "timestamp": 1464192255402,
    "uptime": "23.9h",
    "uptime_in_millis": 86054857,
    "mem": {
      "heap_used": "518.5mb",
      "heap_used_percent": 2,
      "heap_committed": "19.9gb",
      "heap_max": "19.9gb",
      "non_heap_used": "105.7mb",
      "non_heap_committed": "107.3mb",

```

Node on Ubuntu 16.04:

```
"jvm": {
    "timestamp": 1464192290592,
    "uptime": "2.6m",
    "uptime_in_millis": 156539,
    "mem": {
      "heap_used": "389.8mb",
      "heap_used_percent": 1,
      "heap_committed": "19.9gb",
      "heap_max": "19.9gb",
      "non_heap_used": "81.7mb",
      "non_heap_committed": "83.2mb",

```

Even though the jvm stats are relatively the same but that's what I see in my machine:

Ubuntu 16.04: **free -m**

```
              total used free shared buff/cache available
Mem: 60399 1653 58212 8 533 58618
Swap: 0 0 0

```

Ubuntu 14.04: **free -m**

```
             total used free shared buffers cached
Mem: 60400 58669 1731 0 326 35014
-/+ buffers/cache: 23328 37071
Swap: 0 0 0
```

---

<div class="post-metadata">

**Author:** ![maziyar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/maziyar/32/18423_2.png) [@maziyar](https://discuss.elastic.co/u/maziyar)\
**Post date:** [May 25, 2016, 5:19pm UTC](https://discuss.elastic.co/t/memory-confusion-in-ubuntu-16-04/50961/2 "2016-05-25T17:19:18Z")

</div>

Also this is what happens to the heap size in Ubuntu 14.04 compare to Ubuntu 16.04:

Ubuntu 14.04  
[![](https://dl.dropboxusercontent.com/s/n1akwn909skdqii/Screenshot%202016-05-25%2019.16.46.png?dl=0) ](https://dl.dropboxusercontent.com/s/n1akwn909skdqii/Screenshot%202016-05-25%2019.16.46.png?dl=0)

Ubuntu 16.04 (Before 17:30 this node was Ubuntu 14.04)  
[![](https://dl.dropboxusercontent.com/s/92iojo50jyznugo/Screenshot%202016-05-25%2019.18.29.png?dl=0) ](https://dl.dropboxusercontent.com/s/92iojo50jyznugo/Screenshot%202016-05-25%2019.18.29.png?dl=0)

---

<div class="post-metadata">

**Author:** ![YuWatanabe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuwatanabe/32/13259_2.png) [@YuWatanabe](https://discuss.elastic.co/u/YuWatanabe)\
**Post date:** [May 26, 2016, 12:59am UTC](https://discuss.elastic.co/t/memory-confusion-in-ubuntu-16-04/50961/3 "2016-05-26T00:59:49Z")

</div>

I am not a expert but, according to definitive guide, I dont think you need to enable bootstrap.mlockall if the swap space is not mounted.

`bootstrap.mlockall : true`

> **[Heap: Sizing and Swapping | Elasticsearch: The Definitive Guide \[2.x\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html)**

> Finally, if neither approach is possible, you should enable mlockall. file. This allows the JVM to lock its memory and prevent it from being swapped by the OS. In your elasticsearch.yml, set this:

---

<div class="post-metadata">

**Author:** ![maziyar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/maziyar/32/18423_2.png) [@maziyar](https://discuss.elastic.co/u/maziyar)\
**Post date:** [May 26, 2016, 9:43am UTC](https://discuss.elastic.co/t/memory-confusion-in-ubuntu-16-04/50961/4 "2016-05-26T09:43:14Z")

</div>

You are absolutely right! I've been enabling mlockall for years even though the swap was off but this time I didn't even bother to create Swap partition but automaticallyI enabled mlockall in the configs like before. (serious muscle memory right there!)

I commented out mlockall on all the nodes and the warning went away!

Thank you very much 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 10:48pm UTC](https://discuss.elastic.co/t/memory-confusion-in-ubuntu-16-04/50961/5 "2017-07-05T22:48:52Z")

</div>