# Tough Memlock issue in Docker Swarm on Centos **URL:** \ **Category:** Elasticsearch\ **Tags:** docker\ **Created:** [March 25, 2019, 3:33pm UTC](https://discuss.elastic.co/t/tough-memlock-issue-in-docker-swarm-on-centos/173788 "2019-03-25T15:33:07Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Zorlack](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zorlack/32/42730_2.png) [@Zorlack](https://discuss.elastic.co/u/Zorlack)\ **Post date:** [March 25, 2019, 3:33pm UTC](https://discuss.elastic.co/t/tough-memlock-issue-in-docker-swarm-on-centos/173788/1 "2019-03-25T15:33:07Z")
Hello, I'm having trouble lighting up a Elastic Search stack on Docker. The error I'm seeing is this: ``` [2019-03-25T15:10:08,955][WARN][o.e.b.JNANatives] [unknown] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory [2019-03-25T15:10:08,964][WARN][o.e.b.JNANatives] [unknown] This can result in part of the JVM being swapped out. [2019-03-25T15:10:08,964][WARN][o.e.b.JNANatives] [unknown] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536 [2019-03-25T15:10:08,965][WARN][o.e.b.JNANatives] [unknown] These can be adjusted by modifying /etc/security/limits.conf, for example: # allow user 'elasticsearch' mlockall elasticsearch soft memlock unlimited elasticsearch hard memlock unlimited [- - - - SNIP - - - -] ERROR: [1] bootstrap checks failed [1]: memory locking requested for elasticsearch process but memory is not locked [2019-03-25T15:15:17,114][INFO][o.e.n.Node] [yriY5aR] stopping ... [2019-03-25T15:15:17,147][INFO][o.e.n.Node] [yriY5aR] stopped [2019-03-25T15:15:17,147][INFO][o.e.n.Node] [yriY5aR] closing ... [2019-03-25T15:15:17,185][INFO][o.e.n.Node] [yriY5aR] closed [2019-03-25T15:15:17,189][INFO][o.e.x.m.p.NativeController] [yriY5aR] Native controller process has stopped - no new native processes can be started ``` I understand that this means one of two things: Either, the Elasticsearch user doesn't have the correct limits set for memlock, or the Docker user doesn't have the correct limits. To start I created the docker.service overide file: ``` [root@hq-deva-013 ~]# cat /etc/systemd/system/docker.service.d/override.conf [Service] LimitMEMLOCK=infinity ``` After a reboot that seems to have the desired effect as shown by this ``` [root@hq-deva-013 ~]# grep locked /proc/$(ps --no-headers -o pid -C dockerd | tr -d ' ')/limits Max locked memory unlimited unlimited bytes ``` Next, I wanted to make sure that the elasticsearch (1000) user had the right permissions. So I added this to /etc/security/limits.conf `* - memlock unlimited` It seems to work based on this: ``` [root@hq-deva-013 ~]# su - elasticsearch Last login: Mon Mar 25 10:41:11 EDT 2019 on pts/0 [elasticsearch@hq-deva-013 ~]$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 15025 max locked memory (kbytes, -l) unlimited max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 4096 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited ``` Unfortunately I'm still seeing the same behavior. I'm not sure where to look next. Here's the stack configuration I'm using: ``` version: '3.3' services: elasticsearch: image: docker.elastic.co/elasticsearch/elasticsearch:6.6.2 environment: ES_JAVA_OPTS: -Xms512m -Xmx512m bootstrap.memory_lock: 'true' cluster.name: docker-cluster ports: - target: 9200 published: 9200 protocol: tcp mode: host - target: 9300 published: 9300 protocol: tcp mode: host volumes: - esdata1:/usr/share/elasticsearch/data networks: - esnet logging: driver: json-file elasticsearch2: image: docker.elastic.co/elasticsearch/elasticsearch:6.6.2 environment: ES_JAVA_OPTS: -Xms512m -Xmx512m bootstrap.memory_lock: 'true' cluster.name: docker-cluster discovery.zen.ping.unicast.hosts: elasticsearch ports: - target: 9200 published: 9200 protocol: tcp mode: host - target: 9300 published: 9300 protocol: tcp mode: host volumes: - esdata2:/usr/share/elasticsearch/data networks: - esnet logging: driver: json-file networks: esnet: driver: overlay volumes: esdata1: driver: local esdata2: driver: local ``` **Where should I be looking to resolve this? Is this a docker thing? A system thing?** Many thanks! -Z ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [March 25, 2019, 3:44pm UTC](https://discuss.elastic.co/t/tough-memlock-issue-in-docker-swarm-on-centos/173788/2 "2019-03-25T15:44:23Z")
Using memory locking is [a last resort for disabling swapping](https://github.com/elastic/elasticsearch/issues/17605#issuecomment-207149090). It is preferable [simply to disable swap](https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration-memory.html#setup-configuration-memory). Can you do this instead of using `bootstrap.memory_lock`? If you must use `memory_lock`, the [reference docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html) indicate that you need to set some settings on the container. It's not clear from your post, but I suspect you're only setting the `memlock` limits on the host. ---
**Author:** ![Zorlack](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zorlack/32/42730_2.png) [@Zorlack](https://discuss.elastic.co/u/Zorlack)\ **Post date:** [March 25, 2019, 3:53pm UTC](https://discuss.elastic.co/t/tough-memlock-issue-in-docker-swarm-on-centos/173788/3 "2019-03-25T15:53:48Z")
Hi David, Per [Docker](https://docs.docker.com/compose/compose-file/#ulimits), the ulimits configuration section is ignored during Docker Swarm deploy: ``` ulimits: memlock: soft: -1 hard: -1 ``` So I guess the documentation doesn't cover the swarm scenario. I saw the bootstrap.memory\_lock option in the documentation and assumed it was best practice. It seems like I can disregard this completely if I know the Docker hosts have no swap. Is this true? Many thanks! -Z ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [March 25, 2019, 4:05pm UTC](https://discuss.elastic.co/t/tough-memlock-issue-in-docker-swarm-on-centos/173788/4 "2019-03-25T16:05:17Z")
> [@Zorlack](#): > > It seems like I can disregard this completely if I know the Docker hosts have no swap. Is this true? I have not verified this myself, but I have heard that if swap is disabled on the host then it's also disabled on the containers running on that host. ---
**Author:** ![Zorlack](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zorlack/32/42730_2.png) [@Zorlack](https://discuss.elastic.co/u/Zorlack)\ **Post date:** [March 25, 2019, 4:11pm UTC](https://discuss.elastic.co/t/tough-memlock-issue-in-docker-swarm-on-centos/173788/5 "2019-03-25T16:11:02Z")
Thank you for your help @DavidTurner. I will investigate the system VM issue further. Cheers! -Z ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 22, 2019, 4:11pm UTC](https://discuss.elastic.co/t/tough-memlock-issue-in-docker-swarm-on-centos/173788/6 "2019-04-22T16:11:08Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.