Supervisord: memlock problem

hello,

i am building a solution where i have to start native tar.gz deployments with supervisord.
elasticsearch limits are set like this:

elasticsearch   hard    nofile  65536
elasticsearch   hard    memlock unlimited
elasticsearch   hard    nproc   65536
elasticsearch   hard    stack   327680
elasticsearch   soft    nofile  65536
elasticsearch   soft    memlock unlimited
elasticsearch   soft    nproc   65536
elasticsearch   soft    stack   327680

supervisord.conf:

[unix_http_server]
file=/var/run/supervisor/supervisor.sock   ; (the path to the socket file)
;chmod=0700                 ; sockef file mode (default 0700)
;chown=nobody:nogroup       ; socket file uid:gid owner
;username=user              ; (default is no username (open server))
;password=123               ; (default is no password (open server))

;[inet_http_server]         ; inet (TCP) server disabled by default
;port=127.0.0.1:9001        ; (ip_address:port specifier, *:port for all iface)
;username=user              ; (default is no username (open server))
;password=123               ; (default is no password (open server))

[supervisord]
logfile=/var/log/supervisor/supervisord.log  ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB       ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10          ; (num of main logfile rotation backups;default 10)
loglevel=info               ; (log level;default info; others: debug,warn,trace)
pidfile=/var/run/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false              ; (start in foreground if true;default false)
minfds=65536                ; (min. avail startup file descriptors;default 1024)
minprocs=65536                ; (min. avail process descriptors;default 200)

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///var/run/supervisor/supervisor.sock ; use a unix:// URL  for a unix socket

[program:elasticsearch]
user=elasticsearch
group=elasticsearch
command=/app/elasticsearch/6.5.4/bin/elasticsearch
autostart=true
autorestart=true
environment=JAVA_HOME=/app/java/11,MAX_LOCKED_MEMORY=unlimited,LimitMEMLOCK=infinity
stdout_logfile=/app/elasticsearch/6.5.4/logs/elasticsearch.out.log
stderr_logfile=/app/elasticsearch/6.5.4/logs/elasticsearch.err.log

2 of 4 bootstrap checks are failing :frowning: i do not know where to go

elasticsearch log file:

5ef32393) Copyright (c) 2018 Elasticsearch BV
[2019-01-15T18:45:36,892][DEBUG][o.e.a.ActionModule       ] [master1-node1] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2019-01-15T18:45:38,015][INFO ][o.e.d.DiscoveryModule    ] [master1-node1] using discovery type [zen] and host providers [settings]
[2019-01-15T18:45:43,626][INFO ][o.e.n.Node               ] [master1-node1] initialized
[2019-01-15T18:45:43,627][INFO ][o.e.n.Node               ] [master1-node1] starting ...
[2019-01-15T18:45:44,755][INFO ][o.e.t.TransportService   ] [master1-node1] publish_address {10.0.2.15:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}, {192.168.10.101:9300}, {10.0.2.15:9300}
[2019-01-15T18:45:44,831][INFO ][o.e.b.BootstrapChecks    ] [master1-node1] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-01-15T18:45:44,864][ERROR][o.e.b.Bootstrap          ] [master1-node1] node validation exception
[2] bootstrap checks failed
[1]: memory locking requested for elasticsearch process but memory is not locked
[2]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2019-01-15T18:45:44,923][INFO ][o.e.n.Node               ] [master1-node1] stopping ...
[2019-01-15T18:45:44,952][INFO ][o.e.n.Node               ] [master1-node1] stopped
[2019-01-15T18:45:44,953][INFO ][o.e.n.Node               ] [master1-node1] closing ...
[2019-01-15T18:45:45,038][INFO ][o.e.n.Node               ] [master1-node1] closed
[2019-01-15T18:45:45,067][INFO ][o.e.x.m.j.p.NativeController] [master1-node1] Native controller process has stopped - no new native processes can be started
[2019-01-15T18:46:00,911][WARN ][o.e.b.JNANatives         ] [master1-node1] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
[2019-01-15T18:46:00,937][WARN ][o.e.b.JNANatives         ] [master1-node1] This can result in part of the JVM being swapped out.
[2019-01-15T18:46:00,937][WARN ][o.e.b.JNANatives         ] [master1-node1] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
[2019-01-15T18:46:00,938][WARN ][o.e.b.JNANatives         ] [master1-node1] These can be adjusted by modifying /etc/security/limits.conf, for example:
    # allow user 'elasticsearch' mlockall
    elasticsearch soft memlock unlimited
    elasticsearch hard memlock unlimited
[2019-01-15T18:46:00,938][WARN ][o.e.b.JNANatives         ] [master1-node1] If you are logged in interactively, you will have to re-login for the new limits to take effect.
[2019-01-15T18:46:03,482][INFO ][o.e.e.NodeEnvironment    ] [master1-node1] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [33.9gb], net total_space [39.2gb], types [rootfs]
[2019-01-15T18:46:03,498][INFO ][o.e.e.NodeEnvironment    ] [master1-node1] heap size [1015.6mb], compressed ordinary object pointers [true]
[2019-01-15T18:46:03,502][INFO ][o.e.n.Node               ] [master1-node1] node name [master1-node1], node ID [fatSPKAlQN-UtjCIDRskzQ]
[2019-01-15T18:46:03,502][INFO ][o.e.n.Node               ] [master1-node1] version[6.5.4], pid[32059], build[default/tar/d2ef93d/2018-12-17T21:17:40.758843Z], OS[Linux/3.10.0-862.14.4.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/11.0.1/11.0.1+13]
[2019-01-15T18:46:03,503][INFO ][o.e.n.Node               ] [master1-node1] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.qlv4qz7e, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -XX:UseAVX=2, -Des.path.home=/app/elasticsearch/6.5.4, -Des.path.conf=/app/elasticsearch/6.5.4/config, -Des.distribution.flavor=default, -Des.distribution.type=tar]

thx very much
kind regards
Markus

Managed to do it.

added /etc/systemd/system/supervisord.service.d/10-memlock.conf:

[Service]
LimitMEMLOCK=infinity

and to /etc/sysctl.conf

vm.max_map_count=262144

it worked.

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.