[ES 5.0.0-alpha5] Unable to lock JVM Memory: error=12


(AndrzejS) #1

Hi,
I installed ES 5.0.0-alpha 5 on Debian Jessie and cannot lock JVM Memory

I set the below

echo 'elasticsearch soft memlock unlimited' | tee -a /etc/security/limits.conf
echo 'elasticsearch hard memlock unlimited' | tee -a /etc/security/limits.conf
/etc/elasticsearch/elasticsearch.yml bootstrap.memory_lock: true

on elasticsearch account

ulimit -H -l
unlimited

in logs have this:

[2016-09-08 14:13:13,979][WARN ][bootstrap                ] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
[2016-09-08 14:13:13,980][WARN ][bootstrap                ] This can result in part of the JVM being swapped out.
[2016-09-08 14:13:13,981][WARN ][bootstrap                ] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
[2016-09-08 14:13:13,981][WARN ][bootstrap                ] These can be adjusted by modifying /etc/security/limits.conf, for example:
        # allow user 'elasticsearch' mlockall
        elasticsearch soft memlock unlimited
        elasticsearch hard memlock unlimited
[2016-09-08 14:13:13,981][WARN ][bootstrap                ] If you are logged in interactively, you will have to re-login for the new limits to take effect.
[2016-09-08 14:13:14,195][INFO ][node                     ] [] initializing ...
[2016-09-08 14:13:14,323][INFO ][env                      ] [gLqcDCB] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [28.7gb], net total_space [32.8gb], spins? [unknown], types [rootfs]
[2016-09-08 14:13:14,323][INFO ][env                      ] [gLqcDCB] heap size [1.9gb], compressed ordinary object pointers [true]
[2016-09-08 14:13:14,326][INFO ][node                     ] [gLqcDCB] node name [gLqcDCB] derived from node ID; set [node.name] to override
[2016-09-08 14:13:14,341][INFO ][node                     ] [gLqcDCB] version[5.0.0-alpha5], pid[2479], build[d327dd4/2016-08-04T08:59:39.568Z], OS[Linux/3.16.0-4-amd64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_101/25.101-b13]
[2016-09-08 14:13:14,779][INFO ][io.netty.util.internal.PlatformDependent] Your platform does not provide complete low-level API for accessing direct buffers reliably. Unless explicitly requested, heap buffer will always be preferred to avoid potential system unstability.
[2016-09-08 14:13:15,935][INFO ][plugins                  ] [gLqcDCB] loaded module [aggs-matrix-stats]
[2016-09-08 14:13:15,935][INFO ][plugins                  ] [gLqcDCB] loaded module [ingest-common]
[2016-09-08 14:13:15,935][INFO ][plugins                  ] [gLqcDCB] loaded module [lang-expression]
[2016-09-08 14:13:15,936][INFO ][plugins                  ] [gLqcDCB] loaded module [lang-groovy]
[2016-09-08 14:13:15,936][INFO ][plugins                  ] [gLqcDCB] loaded module [lang-mustache]
[2016-09-08 14:13:15,936][INFO ][plugins                  ] [gLqcDCB] loaded module [lang-painless]
[2016-09-08 14:13:15,936][INFO ][plugins                  ] [gLqcDCB] loaded module [percolator]
[2016-09-08 14:13:15,936][INFO ][plugins                  ] [gLqcDCB] loaded module [reindex]
[2016-09-08 14:13:15,936][INFO ][plugins                  ] [gLqcDCB] loaded module [transport-netty3]
[2016-09-08 14:13:15,936][INFO ][plugins                  ] [gLqcDCB] loaded module [transport-netty4]
[2016-09-08 14:13:15,937][INFO ][plugins                  ] [gLqcDCB] no plugins loaded
[2016-09-08 14:13:18,912][INFO ][node                     ] [gLqcDCB] initialized
[2016-09-08 14:13:18,913][INFO ][node                     ] [gLqcDCB] starting ...
[2016-09-08 14:13:19,116][INFO ][transport                ] [gLqcDCB] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2016-09-08 14:13:19,126][WARN ][bootstrap                ] [gLqcDCB] initial heap size [268435456] not equal to maximum heap size [2147483648]; this can cause resize pauses and prevents mlockall from locking the entire heap
[2016-09-08 14:13:19,126][WARN ][bootstrap                ] [gLqcDCB] memory locking requested for elasticsearch process but memory is not locked
[2016-09-08 14:13:19,126][WARN ][bootstrap                ] [gLqcDCB] please set [discovery.zen.minimum_master_nodes] to a majority of the number of master eligible nodes in your cluster
[2016-09-08 14:13:22,259][INFO ][cluster.service          ] [gLqcDCB] new_master {gLqcDCB}{gLqcDCBcTtOgOe_kI4d2LA}{jMA9HkxHRkeDzA7FKorw7g}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2016-09-08 14:13:22,287][INFO ][http                     ] [gLqcDCB] publish_address {127.0.0.1:9200}, bound_addresses {[::]:9200}
[2016-09-08 14:13:22,287][INFO ][node                     ] [gLqcDCB] started
[2016-09-08 14:13:22,306][INFO ][gateway                  ] [gLqcDCB] recovered [0] indices into cluster_state

(Alexander Reelsen) #2

Hey,

did you also configure systemd appropriately? See https://www.elastic.co/guide/en/elasticsearch/reference/5.0/setup-configuration-memory.html#mlockall

--Alex


(AndrzejS) #3

Works but i must enable this: LimitMEMLOCK=infinity
Thx


#4

Hi, I am facing a similar issue. Did you also set MAX_LOCKED_MEMORY in /etc/default/elasticsearch?


(system) #5