Supervisord: memlock problem


i am building a solution where i have to start native tar.gz deployments with supervisord.
elasticsearch limits are set like this:

elasticsearch   hard    nofile  65536
elasticsearch   hard    memlock unlimited
elasticsearch   hard    nproc   65536
elasticsearch   hard    stack   327680
elasticsearch   soft    nofile  65536
elasticsearch   soft    memlock unlimited
elasticsearch   soft    nproc   65536
elasticsearch   soft    stack   327680


file=/var/run/supervisor/supervisor.sock   ; (the path to the socket file)
;chmod=0700                 ; sockef file mode (default 0700)
;chown=nobody:nogroup       ; socket file uid:gid owner
;username=user              ; (default is no username (open server))
;password=123               ; (default is no password (open server))

;[inet_http_server]         ; inet (TCP) server disabled by default
;port=        ; (ip_address:port specifier, *:port for all iface)
;username=user              ; (default is no username (open server))
;password=123               ; (default is no password (open server))

logfile=/var/log/supervisor/supervisord.log  ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB       ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10          ; (num of main logfile rotation backups;default 10)
loglevel=info               ; (log level;default info; others: debug,warn,trace)
pidfile=/var/run/ ; (supervisord pidfile;default
nodaemon=false              ; (start in foreground if true;default false)
minfds=65536                ; (min. avail startup file descriptors;default 1024)
minprocs=65536                ; (min. avail process descriptors;default 200)

supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

serverurl=unix:///var/run/supervisor/supervisor.sock ; use a unix:// URL  for a unix socket


2 of 4 bootstrap checks are failing :frowning: i do not know where to go

elasticsearch log file:

5ef32393) Copyright (c) 2018 Elasticsearch BV
[2019-01-15T18:45:36,892][DEBUG][o.e.a.ActionModule       ] [master1-node1] Using REST wrapper from plugin
[2019-01-15T18:45:38,015][INFO ][o.e.d.DiscoveryModule    ] [master1-node1] using discovery type [zen] and host providers [settings]
[2019-01-15T18:45:43,626][INFO ][o.e.n.Node               ] [master1-node1] initialized
[2019-01-15T18:45:43,627][INFO ][o.e.n.Node               ] [master1-node1] starting ...
[2019-01-15T18:45:44,755][INFO ][o.e.t.TransportService   ] [master1-node1] publish_address {}, bound_addresses {[::1]:9300}, {}, {}, {}
[2019-01-15T18:45:44,831][INFO ][o.e.b.BootstrapChecks    ] [master1-node1] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-01-15T18:45:44,864][ERROR][o.e.b.Bootstrap          ] [master1-node1] node validation exception
[2] bootstrap checks failed
[1]: memory locking requested for elasticsearch process but memory is not locked
[2]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2019-01-15T18:45:44,923][INFO ][o.e.n.Node               ] [master1-node1] stopping ...
[2019-01-15T18:45:44,952][INFO ][o.e.n.Node               ] [master1-node1] stopped
[2019-01-15T18:45:44,953][INFO ][o.e.n.Node               ] [master1-node1] closing ...
[2019-01-15T18:45:45,038][INFO ][o.e.n.Node               ] [master1-node1] closed
[2019-01-15T18:45:45,067][INFO ][o.e.x.m.j.p.NativeController] [master1-node1] Native controller process has stopped - no new native processes can be started
[2019-01-15T18:46:00,911][WARN ][o.e.b.JNANatives         ] [master1-node1] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
[2019-01-15T18:46:00,937][WARN ][o.e.b.JNANatives         ] [master1-node1] This can result in part of the JVM being swapped out.
[2019-01-15T18:46:00,937][WARN ][o.e.b.JNANatives         ] [master1-node1] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
[2019-01-15T18:46:00,938][WARN ][o.e.b.JNANatives         ] [master1-node1] These can be adjusted by modifying /etc/security/limits.conf, for example:
    # allow user 'elasticsearch' mlockall
    elasticsearch soft memlock unlimited
    elasticsearch hard memlock unlimited
[2019-01-15T18:46:00,938][WARN ][o.e.b.JNANatives         ] [master1-node1] If you are logged in interactively, you will have to re-login for the new limits to take effect.
[2019-01-15T18:46:03,482][INFO ][o.e.e.NodeEnvironment    ] [master1-node1] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [33.9gb], net total_space [39.2gb], types [rootfs]
[2019-01-15T18:46:03,498][INFO ][o.e.e.NodeEnvironment    ] [master1-node1] heap size [1015.6mb], compressed ordinary object pointers [true]
[2019-01-15T18:46:03,502][INFO ][o.e.n.Node               ] [master1-node1] node name [master1-node1], node ID [fatSPKAlQN-UtjCIDRskzQ]
[2019-01-15T18:46:03,502][INFO ][o.e.n.Node               ] [master1-node1] version[6.5.4], pid[32059], build[default/tar/d2ef93d/2018-12-17T21:17:40.758843Z], OS[Linux/3.10.0-862.14.4.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/11.0.1/11.0.1+13]
[2019-01-15T18:46:03,503][INFO ][o.e.n.Node               ] [master1-node1] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true,, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -XX:UseAVX=2, -Des.path.home=/app/elasticsearch/6.5.4, -Des.path.conf=/app/elasticsearch/6.5.4/config, -Des.distribution.flavor=default, -Des.distribution.type=tar]

thx very much
kind regards

Managed to do it.

added /etc/systemd/system/supervisord.service.d/10-memlock.conf:


and to /etc/sysctl.conf


it worked.

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.