Tough Memlock issue in Docker Swarm on Centos

Elastic Stack Elasticsearch
1

Hello,

I'm having trouble lighting up a Elastic Search stack on Docker.

The error I'm seeing is this:

[2019-03-25T15:10:08,955][WARN ][o.e.b.JNANatives         ] [unknown] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
[2019-03-25T15:10:08,964][WARN ][o.e.b.JNANatives         ] [unknown] This can result in part of the JVM being swapped out.
[2019-03-25T15:10:08,964][WARN ][o.e.b.JNANatives         ] [unknown] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
[2019-03-25T15:10:08,965][WARN ][o.e.b.JNANatives         ] [unknown] These can be adjusted by modifying /etc/security/limits.conf, for example:
        # allow user 'elasticsearch' mlockall
        elasticsearch soft memlock unlimited
        elasticsearch hard memlock unlimited
[ - - - - SNIP - - - -]
ERROR: [1] bootstrap checks failed
[1]: memory locking requested for elasticsearch process but memory is not locked
[2019-03-25T15:15:17,114][INFO ][o.e.n.Node               ] [yriY5aR] stopping ...
[2019-03-25T15:15:17,147][INFO ][o.e.n.Node               ] [yriY5aR] stopped
[2019-03-25T15:15:17,147][INFO ][o.e.n.Node               ] [yriY5aR] closing ...
[2019-03-25T15:15:17,185][INFO ][o.e.n.Node               ] [yriY5aR] closed
[2019-03-25T15:15:17,189][INFO ][o.e.x.m.p.NativeController] [yriY5aR] Native controller process has stopped - no new native processes can be started

I understand that this means one of two things: Either, the Elasticsearch user doesn't have the correct limits set for memlock, or the Docker user doesn't have the correct limits.

To start I created the docker.service overide file:

[root@hq-deva-013 ~]# cat /etc/systemd/system/docker.service.d/override.conf
[Service]
LimitMEMLOCK=infinity

After a reboot that seems to have the desired effect as shown by this

[root@hq-deva-013 ~]# grep locked /proc/$(ps --no-headers -o pid -C dockerd | tr -d ' ')/limits
Max locked memory         unlimited            unlimited            bytes

Next, I wanted to make sure that the elasticsearch (1000) user had the right permissions. So I added this to /etc/security/limits.conf

* - memlock unlimited

It seems to work based on this:

[root@hq-deva-013 ~]# su - elasticsearch
Last login: Mon Mar 25 10:41:11 EDT 2019 on pts/0
[elasticsearch@hq-deva-013 ~]$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 15025
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 4096
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Unfortunately I'm still seeing the same behavior.

I'm not sure where to look next. Here's the stack configuration I'm using:

version: '3.3'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.6.2
    environment:
      ES_JAVA_OPTS: -Xms512m -Xmx512m
      bootstrap.memory_lock: 'true'
      cluster.name: docker-cluster
    ports:
      - target: 9200
        published: 9200
        protocol: tcp
        mode: host
      - target: 9300
        published: 9300
        protocol: tcp
        mode: host 
    volumes:
     - esdata1:/usr/share/elasticsearch/data
    networks:
     - esnet
    logging:
      driver: json-file
  elasticsearch2:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.6.2
    environment:
      ES_JAVA_OPTS: -Xms512m -Xmx512m
      bootstrap.memory_lock: 'true'
      cluster.name: docker-cluster
      discovery.zen.ping.unicast.hosts: elasticsearch
    ports:
      - target: 9200
        published: 9200
        protocol: tcp
        mode: host
      - target: 9300
        published: 9300
        protocol: tcp
        mode: host 
    volumes:
     - esdata2:/usr/share/elasticsearch/data
    networks:
     - esnet
    logging:
      driver: json-file
networks:
  esnet:
    driver: overlay
volumes:
  esdata1:
    driver: local
  esdata2:
    driver: local

Where should I be looking to resolve this? Is this a docker thing? A system thing?

Many thanks!

-Z

2

Using memory locking is a last resort for disabling swapping. It is preferable simply to disable swap. Can you do this instead of using bootstrap.memory_lock?

If you must use memory_lock, the reference docs indicate that you need to set some settings on the container. It's not clear from your post, but I suspect you're only setting the memlock limits on the host.

3

Hi David,

Per Docker, the ulimits configuration section is ignored during Docker Swarm deploy:

ulimits:
  memlock:
    soft: -1
    hard: -1

So I guess the documentation doesn't cover the swarm scenario.

I saw the bootstrap.memory_lock option in the documentation and assumed it was best practice.

It seems like I can disregard this completely if I know the Docker hosts have no swap. Is this true?

Many thanks!

-Z

4

I have not verified this myself, but I have heard that if swap is disabled on the host then it's also disabled on the containers running on that host.

5

Thank you for your help @DavidTurner. I will investigate the system VM issue further.

Cheers!

-Z

1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.