hello,
i am building a solution where i have to start native tar.gz deployments with supervisord.
elasticsearch limits are set like this:
elasticsearch hard nofile 65536
elasticsearch hard memlock unlimited
elasticsearch hard nproc 65536
elasticsearch hard stack 327680
elasticsearch soft nofile 65536
elasticsearch soft memlock unlimited
elasticsearch soft nproc 65536
elasticsearch soft stack 327680
supervisord.conf:
[unix_http_server]
file=/var/run/supervisor/supervisor.sock ; (the path to the socket file)
;chmod=0700 ; sockef file mode (default 0700)
;chown=nobody:nogroup ; socket file uid:gid owner
;username=user ; (default is no username (open server))
;password=123 ; (default is no password (open server))
;[inet_http_server] ; inet (TCP) server disabled by default
;port=127.0.0.1:9001 ; (ip_address:port specifier, *:port for all iface)
;username=user ; (default is no username (open server))
;password=123 ; (default is no password (open server))
[supervisord]
logfile=/var/log/supervisor/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10 ; (num of main logfile rotation backups;default 10)
loglevel=info ; (log level;default info; others: debug,warn,trace)
pidfile=/var/run/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false ; (start in foreground if true;default false)
minfds=65536 ; (min. avail startup file descriptors;default 1024)
minprocs=65536 ; (min. avail process descriptors;default 200)
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///var/run/supervisor/supervisor.sock ; use a unix:// URL for a unix socket
[program:elasticsearch]
user=elasticsearch
group=elasticsearch
command=/app/elasticsearch/6.5.4/bin/elasticsearch
autostart=true
autorestart=true
environment=JAVA_HOME=/app/java/11,MAX_LOCKED_MEMORY=unlimited,LimitMEMLOCK=infinity
stdout_logfile=/app/elasticsearch/6.5.4/logs/elasticsearch.out.log
stderr_logfile=/app/elasticsearch/6.5.4/logs/elasticsearch.err.log
2 of 4 bootstrap checks are failing i do not know where to go
elasticsearch log file:
5ef32393) Copyright (c) 2018 Elasticsearch BV
[2019-01-15T18:45:36,892][DEBUG][o.e.a.ActionModule ] [master1-node1] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2019-01-15T18:45:38,015][INFO ][o.e.d.DiscoveryModule ] [master1-node1] using discovery type [zen] and host providers [settings]
[2019-01-15T18:45:43,626][INFO ][o.e.n.Node ] [master1-node1] initialized
[2019-01-15T18:45:43,627][INFO ][o.e.n.Node ] [master1-node1] starting ...
[2019-01-15T18:45:44,755][INFO ][o.e.t.TransportService ] [master1-node1] publish_address {10.0.2.15:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}, {192.168.10.101:9300}, {10.0.2.15:9300}
[2019-01-15T18:45:44,831][INFO ][o.e.b.BootstrapChecks ] [master1-node1] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-01-15T18:45:44,864][ERROR][o.e.b.Bootstrap ] [master1-node1] node validation exception
[2] bootstrap checks failed
[1]: memory locking requested for elasticsearch process but memory is not locked
[2]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2019-01-15T18:45:44,923][INFO ][o.e.n.Node ] [master1-node1] stopping ...
[2019-01-15T18:45:44,952][INFO ][o.e.n.Node ] [master1-node1] stopped
[2019-01-15T18:45:44,953][INFO ][o.e.n.Node ] [master1-node1] closing ...
[2019-01-15T18:45:45,038][INFO ][o.e.n.Node ] [master1-node1] closed
[2019-01-15T18:45:45,067][INFO ][o.e.x.m.j.p.NativeController] [master1-node1] Native controller process has stopped - no new native processes can be started
[2019-01-15T18:46:00,911][WARN ][o.e.b.JNANatives ] [master1-node1] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
[2019-01-15T18:46:00,937][WARN ][o.e.b.JNANatives ] [master1-node1] This can result in part of the JVM being swapped out.
[2019-01-15T18:46:00,937][WARN ][o.e.b.JNANatives ] [master1-node1] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
[2019-01-15T18:46:00,938][WARN ][o.e.b.JNANatives ] [master1-node1] These can be adjusted by modifying /etc/security/limits.conf, for example:
# allow user 'elasticsearch' mlockall
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited
[2019-01-15T18:46:00,938][WARN ][o.e.b.JNANatives ] [master1-node1] If you are logged in interactively, you will have to re-login for the new limits to take effect.
[2019-01-15T18:46:03,482][INFO ][o.e.e.NodeEnvironment ] [master1-node1] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [33.9gb], net total_space [39.2gb], types [rootfs]
[2019-01-15T18:46:03,498][INFO ][o.e.e.NodeEnvironment ] [master1-node1] heap size [1015.6mb], compressed ordinary object pointers [true]
[2019-01-15T18:46:03,502][INFO ][o.e.n.Node ] [master1-node1] node name [master1-node1], node ID [fatSPKAlQN-UtjCIDRskzQ]
[2019-01-15T18:46:03,502][INFO ][o.e.n.Node ] [master1-node1] version[6.5.4], pid[32059], build[default/tar/d2ef93d/2018-12-17T21:17:40.758843Z], OS[Linux/3.10.0-862.14.4.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/11.0.1/11.0.1+13]
[2019-01-15T18:46:03,503][INFO ][o.e.n.Node ] [master1-node1] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.qlv4qz7e, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -XX:UseAVX=2, -Des.path.home=/app/elasticsearch/6.5.4, -Des.path.conf=/app/elasticsearch/6.5.4/config, -Des.distribution.flavor=default, -Des.distribution.type=tar]
thx very much
kind regards
Markus