there's the logs that the down node print
] [10.10.2.221_node2] loaded plugin [analysis-ik]
[2019-01-07T19:22:25,406][INFO ][o.e.d.DiscoveryModule ] [10.10.2.221_node2] using discovery type [zen]
[2019-01-07T19:22:27,699][INFO ][o.e.n.Node ] [10.10.2.221_node2] initialized
[2019-01-07T19:22:27,700][INFO ][o.e.n.Node ] [10.10.2.221_node2] starting ...
[2019-01-07T19:22:28,426][INFO ][o.e.t.TransportService ] [10.10.2.221_node2] publish_address {10.10.2.221:9301}, bound_addresses {10.10.2.221:9301}
[2019-01-07T19:22:28,440][INFO ][o.e.b.BootstrapChecks ] [10.10.2.221_node2] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2019-01-07T19:22:28,443][ERROR][o.e.b.Bootstrap ] [10.10.2.221_node2] node validation exception
[2] bootstrap checks failed
[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
[2]: memory locking requested for elasticsearch process but memory is not locked
[2019-01-07T19:22:28,446][INFO ][o.e.n.Node ] [10.10.2.221_node2] stopping ...
[2019-01-07T19:22:28,657][INFO ][o.e.n.Node ] [10.10.2.221_node2] stopped
[2019-01-07T19:22:28,657][INFO ][o.e.n.Node ] [10.10.2.221_node2] closing ...
[2019-01-07T19:22:28,668][INFO ][o.e.n.Node ] [10.10.2.221_node2] closed
[2019-01-07T19:23:44,369][WARN ][o.e.b.JNANatives ] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
[2019-01-07T19:23:44,372][WARN ][o.e.b.JNANatives ] This can result in part of the JVM being swapped out.
[2019-01-07T19:23:44,372][WARN ][o.e.b.JNANatives ] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
[2019-01-07T19:23:44,372][WARN ][o.e.b.JNANatives ] These can be adjusted by modifying /etc/security/limits.conf, for example:
# allow user 'wsn' mlockall
wsn soft memlock unlimited
wsn hard memlock unlimited
[2019-01-07T19:23:44,372][WARN ][o.e.b.JNANatives ] If you are logged in interactively, you will have to re-login for the new limits to take effect.
[2019-01-07T19:23:44,550][INFO ][o.e.n.Node ] [10.10.2.221_node2] initializing ...
[2019-01-07T19:23:44,673][INFO ][o.e.e.NodeEnvironment ] [10.10.2.221_node2] using [1] data paths, mounts [[/ (/dev/mapper/ubuntu--vg-root)]], net usable_space [2.2tb], net total_space [7.4tb], spins? [possibly], types [ext4]
[2019-01-07T19:23:44,674][INFO ][o.e.e.NodeEnvironment ] [10.10.2.221_node2] heap size [30.8gb], compressed ordinary object pointers [true]
[2019-01-07T19:23:44,964][INFO ][o.e.n.Node ] [10.10.2.221_node2] node name [10.10.2.221_node2], node ID [tyAyl2r1SJi4TULBmUZfMg]
[2019-01-07T19:23:44,965][INFO ][o.e.n.Node ] [10.10.2.221_node2] version[5.6.4], pid[31447], build[8bbedf5/2017-10-31T18:55:38.105Z], OS[Linux/4.4.0-130-generic/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_171/25.171-b11]
[2019-01-07T19:23:44,965][INFO ][o.e.n.Node ] [10.10.2.221_node2] JVM arguments [-Xms31g, -Xmx31g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/home/wsn/elasticsearch-5.6.4]
[2019-01-07T19:23:46,270][INFO ][o.e.p.PluginsService ] [10.10.2.221_node2] loaded module [aggs-matrix-stats]
[2019-01-07T19:23:46,270][INFO ][o.e.p.PluginsService ] [10.10.2.221_node2] loaded module [ingest-common]
[2019-01-07T19:23:46,270][INFO ][o.e.p.PluginsService ] [10.10.2.221_node2] loaded module [lang-expression]
[2019-01-07T19:23:46,270][INFO ][o.e.p.PluginsService ] [10.10.2.221_node2] loaded module [lang-groovy]
[2019-01-07T19:23:46,270][INFO ][o.e.p.PluginsService ] [10.10.2.221_node2] loaded module [lang-mustache]
[2019-01-07T19:23:46,271][INFO ][o.e.p.PluginsService ] [10.10.2.221_node2] loaded module [lang-painless]
[2019-01-07T19:23:46,271][INFO ][o.e.p.PluginsService ] [10.10.2.221_node2] loaded module [parent-join]
[2019-01-07T19:23:46,271][INFO ][o.e.p.PluginsService ] [10.10.2.221_node2] loaded module [percolator]
[2019-01-07T19:23:46,271][INFO ][o.e.p.PluginsService ] [10.10.2.221_node2] loaded module [reindex]
[2019-01-07T19:23:46,271][INFO ][o.e.p.PluginsService ] [10.10.2.221_node2] loaded module [transport-netty3]
[2019-01-07T19:23:46,271][INFO ][o.e.p.PluginsService ] [10.10.2.221_node2] loaded module [transport-netty4]
[2019-01-07T19:23:46,271][INFO ][o.e.p.PluginsService ] [10.10.2.221_node2] loaded plugin [analysis-ik]
[2019-01-07T19:23:49,801][INFO ][o.e.d.DiscoveryModule ] [10.10.2.221_node2] using discovery type [zen]
[2019-01-07T19:23:51,927][INFO ][o.e.n.Node ] [10.10.2.221_node2] initialized
[2019-01-07T19:23:51,927][INFO ][o.e.n.Node ] [10.10.2.221_node2] starting ...
[2019-01-07T19:23:52,725][INFO ][o.e.t.TransportService ] [10.10.2.221_node2] publish_address {10.10.2.221:9301}, bound_addresses {10.10.2.221:9301}
[2019-01-07T19:23:52,745][INFO ][o.e.b.BootstrapChecks ] [10.10.2.221_node2] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2019-01-07T19:23:52,750][ERROR][o.e.b.Bootstrap ] [10.10.2.221_node2] node validation exception
[2] bootstrap checks failed
[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
[2]: memory locking requested for elasticsearch process but memory is not locked
[2019-01-07T19:23:52,753][INFO ][o.e.n.Node ] [10.10.2.221_node2] stopping ...
[2019-01-07T19:23:52,940][INFO ][o.e.n.Node ] [10.10.2.221_node2] stopped
[2019-01-07T19:23:52,941][INFO ][o.e.n.Node ] [10.10.2.221_node2] closing ...
[2019-01-07T19:23:52,954][INFO ][o.e.n.Node ] [10.10.2.221_node2] closed
the /etc/security/limits.conf content is as follow:
* soft nofile 65536
* hard nofile 131072
* soft nproc 2048
* hard nproc 4096
* soft memlock unlimited
* hard memlock unlimited
the memlock is already set unlimited and the [soft nofile ] is 65536, but i don't known why this configuration didn't work
thanks
iamazy