Why does es node auto down itself

there's the logs that the down node print

] [10.10.2.221_node2] loaded plugin [analysis-ik]
[2019-01-07T19:22:25,406][INFO ][o.e.d.DiscoveryModule    ] [10.10.2.221_node2] using discovery type [zen]
[2019-01-07T19:22:27,699][INFO ][o.e.n.Node               ] [10.10.2.221_node2] initialized
[2019-01-07T19:22:27,700][INFO ][o.e.n.Node               ] [10.10.2.221_node2] starting ...
[2019-01-07T19:22:28,426][INFO ][o.e.t.TransportService   ] [10.10.2.221_node2] publish_address {10.10.2.221:9301}, bound_addresses {10.10.2.221:9301}
[2019-01-07T19:22:28,440][INFO ][o.e.b.BootstrapChecks    ] [10.10.2.221_node2] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2019-01-07T19:22:28,443][ERROR][o.e.b.Bootstrap          ] [10.10.2.221_node2] node validation exception
[2] bootstrap checks failed
[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
[2]: memory locking requested for elasticsearch process but memory is not locked
[2019-01-07T19:22:28,446][INFO ][o.e.n.Node               ] [10.10.2.221_node2] stopping ...
[2019-01-07T19:22:28,657][INFO ][o.e.n.Node               ] [10.10.2.221_node2] stopped
[2019-01-07T19:22:28,657][INFO ][o.e.n.Node               ] [10.10.2.221_node2] closing ...
[2019-01-07T19:22:28,668][INFO ][o.e.n.Node               ] [10.10.2.221_node2] closed
[2019-01-07T19:23:44,369][WARN ][o.e.b.JNANatives         ] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
[2019-01-07T19:23:44,372][WARN ][o.e.b.JNANatives         ] This can result in part of the JVM being swapped out.
[2019-01-07T19:23:44,372][WARN ][o.e.b.JNANatives         ] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
[2019-01-07T19:23:44,372][WARN ][o.e.b.JNANatives         ] These can be adjusted by modifying /etc/security/limits.conf, for example: 
	# allow user 'wsn' mlockall
	wsn soft memlock unlimited
	wsn hard memlock unlimited
[2019-01-07T19:23:44,372][WARN ][o.e.b.JNANatives         ] If you are logged in interactively, you will have to re-login for the new limits to take effect.
[2019-01-07T19:23:44,550][INFO ][o.e.n.Node               ] [10.10.2.221_node2] initializing ...
[2019-01-07T19:23:44,673][INFO ][o.e.e.NodeEnvironment    ] [10.10.2.221_node2] using [1] data paths, mounts [[/ (/dev/mapper/ubuntu--vg-root)]], net usable_space [2.2tb], net total_space [7.4tb], spins? [possibly], types [ext4]
[2019-01-07T19:23:44,674][INFO ][o.e.e.NodeEnvironment    ] [10.10.2.221_node2] heap size [30.8gb], compressed ordinary object pointers [true]
[2019-01-07T19:23:44,964][INFO ][o.e.n.Node               ] [10.10.2.221_node2] node name [10.10.2.221_node2], node ID [tyAyl2r1SJi4TULBmUZfMg]
[2019-01-07T19:23:44,965][INFO ][o.e.n.Node               ] [10.10.2.221_node2] version[5.6.4], pid[31447], build[8bbedf5/2017-10-31T18:55:38.105Z], OS[Linux/4.4.0-130-generic/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_171/25.171-b11]
[2019-01-07T19:23:44,965][INFO ][o.e.n.Node               ] [10.10.2.221_node2] JVM arguments [-Xms31g, -Xmx31g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/home/wsn/elasticsearch-5.6.4]
[2019-01-07T19:23:46,270][INFO ][o.e.p.PluginsService     ] [10.10.2.221_node2] loaded module [aggs-matrix-stats]
[2019-01-07T19:23:46,270][INFO ][o.e.p.PluginsService     ] [10.10.2.221_node2] loaded module [ingest-common]
[2019-01-07T19:23:46,270][INFO ][o.e.p.PluginsService     ] [10.10.2.221_node2] loaded module [lang-expression]
[2019-01-07T19:23:46,270][INFO ][o.e.p.PluginsService     ] [10.10.2.221_node2] loaded module [lang-groovy]
[2019-01-07T19:23:46,270][INFO ][o.e.p.PluginsService     ] [10.10.2.221_node2] loaded module [lang-mustache]
[2019-01-07T19:23:46,271][INFO ][o.e.p.PluginsService     ] [10.10.2.221_node2] loaded module [lang-painless]
[2019-01-07T19:23:46,271][INFO ][o.e.p.PluginsService     ] [10.10.2.221_node2] loaded module [parent-join]
[2019-01-07T19:23:46,271][INFO ][o.e.p.PluginsService     ] [10.10.2.221_node2] loaded module [percolator]
[2019-01-07T19:23:46,271][INFO ][o.e.p.PluginsService     ] [10.10.2.221_node2] loaded module [reindex]
[2019-01-07T19:23:46,271][INFO ][o.e.p.PluginsService     ] [10.10.2.221_node2] loaded module [transport-netty3]
[2019-01-07T19:23:46,271][INFO ][o.e.p.PluginsService     ] [10.10.2.221_node2] loaded module [transport-netty4]
[2019-01-07T19:23:46,271][INFO ][o.e.p.PluginsService     ] [10.10.2.221_node2] loaded plugin [analysis-ik]
[2019-01-07T19:23:49,801][INFO ][o.e.d.DiscoveryModule    ] [10.10.2.221_node2] using discovery type [zen]
[2019-01-07T19:23:51,927][INFO ][o.e.n.Node               ] [10.10.2.221_node2] initialized
[2019-01-07T19:23:51,927][INFO ][o.e.n.Node               ] [10.10.2.221_node2] starting ...
[2019-01-07T19:23:52,725][INFO ][o.e.t.TransportService   ] [10.10.2.221_node2] publish_address {10.10.2.221:9301}, bound_addresses {10.10.2.221:9301}
[2019-01-07T19:23:52,745][INFO ][o.e.b.BootstrapChecks    ] [10.10.2.221_node2] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2019-01-07T19:23:52,750][ERROR][o.e.b.Bootstrap          ] [10.10.2.221_node2] node validation exception
[2] bootstrap checks failed
[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
[2]: memory locking requested for elasticsearch process but memory is not locked
[2019-01-07T19:23:52,753][INFO ][o.e.n.Node               ] [10.10.2.221_node2] stopping ...
[2019-01-07T19:23:52,940][INFO ][o.e.n.Node               ] [10.10.2.221_node2] stopped
[2019-01-07T19:23:52,941][INFO ][o.e.n.Node               ] [10.10.2.221_node2] closing ...
[2019-01-07T19:23:52,954][INFO ][o.e.n.Node               ] [10.10.2.221_node2] closed

the /etc/security/limits.conf content is as follow:

* soft nofile 65536
* hard nofile 131072
* soft nproc 2048
* hard nproc 4096
* soft memlock unlimited
* hard memlock unlimited

the memlock is already set unlimited and the [soft nofile ] is 65536, but i don't known why this configuration didn't work

thanks
iamazy

It seems, that despite the settings being configured, they are not applied properly. Either you may need to relogin (not sure how you start Elasticsearch) or your operating system has another mechanism of setting these and may potentially overwrite or ignore your configuration.

See https://www.elastic.co/guide/en/elasticsearch/reference/6.5/setup-configuration-memory.html#bootstrap-memory_lock for more configuration options and https://www.elastic.co/guide/en/elasticsearch/reference/6.5/file-descriptors.html for more infos about file descriptors.

--Alex

i will try it
Thanks for the response
iamazy

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.