# Why does es node auto down itself

**URL:** <https://discuss.elastic.co/t/why-does-es-node-auto-down-itself/163194>\
**Category:** Elasticsearch\
**Created:** [January 7, 2019, 11:31am UTC](https://discuss.elastic.co/t/why-does-es-node-auto-down-itself/163194 "2019-01-07T11:31:50Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![iamazy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iamazy/32/36501_2.png) [@iamazy](https://discuss.elastic.co/u/iamazy)\
**Post date:** [January 7, 2019, 11:31am UTC](https://discuss.elastic.co/t/why-does-es-node-auto-down-itself/163194/1 "2019-01-07T11:31:50Z")

</div>

there's the logs that the down node print

```auto
] [10.10.2.221_node2] loaded plugin [analysis-ik]
[2019-01-07T19:22:25,406][INFO][o.e.d.DiscoveryModule] [10.10.2.221_node2] using discovery type [zen]
[2019-01-07T19:22:27,699][INFO][o.e.n.Node] [10.10.2.221_node2] initialized
[2019-01-07T19:22:27,700][INFO][o.e.n.Node] [10.10.2.221_node2] starting ...
[2019-01-07T19:22:28,426][INFO][o.e.t.TransportService] [10.10.2.221_node2] publish_address {10.10.2.221:9301}, bound_addresses {10.10.2.221:9301}
[2019-01-07T19:22:28,440][INFO][o.e.b.BootstrapChecks] [10.10.2.221_node2] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2019-01-07T19:22:28,443][ERROR][o.e.b.Bootstrap] [10.10.2.221_node2] node validation exception
[2] bootstrap checks failed
[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
[2]: memory locking requested for elasticsearch process but memory is not locked
[2019-01-07T19:22:28,446][INFO][o.e.n.Node] [10.10.2.221_node2] stopping ...
[2019-01-07T19:22:28,657][INFO][o.e.n.Node] [10.10.2.221_node2] stopped
[2019-01-07T19:22:28,657][INFO][o.e.n.Node] [10.10.2.221_node2] closing ...
[2019-01-07T19:22:28,668][INFO][o.e.n.Node] [10.10.2.221_node2] closed
[2019-01-07T19:23:44,369][WARN][o.e.b.JNANatives] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
[2019-01-07T19:23:44,372][WARN][o.e.b.JNANatives] This can result in part of the JVM being swapped out.
[2019-01-07T19:23:44,372][WARN][o.e.b.JNANatives] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
[2019-01-07T19:23:44,372][WARN][o.e.b.JNANatives] These can be adjusted by modifying /etc/security/limits.conf, for example: 
	# allow user 'wsn' mlockall
	wsn soft memlock unlimited
	wsn hard memlock unlimited
[2019-01-07T19:23:44,372][WARN][o.e.b.JNANatives] If you are logged in interactively, you will have to re-login for the new limits to take effect.
[2019-01-07T19:23:44,550][INFO][o.e.n.Node] [10.10.2.221_node2] initializing ...
[2019-01-07T19:23:44,673][INFO][o.e.e.NodeEnvironment] [10.10.2.221_node2] using [1] data paths, mounts [[/ (/dev/mapper/ubuntu--vg-root)]], net usable_space [2.2tb], net total_space [7.4tb], spins? [possibly], types [ext4]
[2019-01-07T19:23:44,674][INFO][o.e.e.NodeEnvironment] [10.10.2.221_node2] heap size [30.8gb], compressed ordinary object pointers [true]
[2019-01-07T19:23:44,964][INFO][o.e.n.Node] [10.10.2.221_node2] node name [10.10.2.221_node2], node ID [tyAyl2r1SJi4TULBmUZfMg]
[2019-01-07T19:23:44,965][INFO][o.e.n.Node] [10.10.2.221_node2] version[5.6.4], pid[31447], build[8bbedf5/2017-10-31T18:55:38.105Z], OS[Linux/4.4.0-130-generic/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_171/25.171-b11]
[2019-01-07T19:23:44,965][INFO][o.e.n.Node] [10.10.2.221_node2] JVM arguments [-Xms31g, -Xmx31g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/home/wsn/elasticsearch-5.6.4]
[2019-01-07T19:23:46,270][INFO][o.e.p.PluginsService] [10.10.2.221_node2] loaded module [aggs-matrix-stats]
[2019-01-07T19:23:46,270][INFO][o.e.p.PluginsService] [10.10.2.221_node2] loaded module [ingest-common]
[2019-01-07T19:23:46,270][INFO][o.e.p.PluginsService] [10.10.2.221_node2] loaded module [lang-expression]
[2019-01-07T19:23:46,270][INFO][o.e.p.PluginsService] [10.10.2.221_node2] loaded module [lang-groovy]
[2019-01-07T19:23:46,270][INFO][o.e.p.PluginsService] [10.10.2.221_node2] loaded module [lang-mustache]
[2019-01-07T19:23:46,271][INFO][o.e.p.PluginsService] [10.10.2.221_node2] loaded module [lang-painless]
[2019-01-07T19:23:46,271][INFO][o.e.p.PluginsService] [10.10.2.221_node2] loaded module [parent-join]
[2019-01-07T19:23:46,271][INFO][o.e.p.PluginsService] [10.10.2.221_node2] loaded module [percolator]
[2019-01-07T19:23:46,271][INFO][o.e.p.PluginsService] [10.10.2.221_node2] loaded module [reindex]
[2019-01-07T19:23:46,271][INFO][o.e.p.PluginsService] [10.10.2.221_node2] loaded module [transport-netty3]
[2019-01-07T19:23:46,271][INFO][o.e.p.PluginsService] [10.10.2.221_node2] loaded module [transport-netty4]
[2019-01-07T19:23:46,271][INFO][o.e.p.PluginsService] [10.10.2.221_node2] loaded plugin [analysis-ik]
[2019-01-07T19:23:49,801][INFO][o.e.d.DiscoveryModule] [10.10.2.221_node2] using discovery type [zen]
[2019-01-07T19:23:51,927][INFO][o.e.n.Node] [10.10.2.221_node2] initialized
[2019-01-07T19:23:51,927][INFO][o.e.n.Node] [10.10.2.221_node2] starting ...
[2019-01-07T19:23:52,725][INFO][o.e.t.TransportService] [10.10.2.221_node2] publish_address {10.10.2.221:9301}, bound_addresses {10.10.2.221:9301}
[2019-01-07T19:23:52,745][INFO][o.e.b.BootstrapChecks] [10.10.2.221_node2] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2019-01-07T19:23:52,750][ERROR][o.e.b.Bootstrap] [10.10.2.221_node2] node validation exception
[2] bootstrap checks failed
[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
[2]: memory locking requested for elasticsearch process but memory is not locked
[2019-01-07T19:23:52,753][INFO][o.e.n.Node] [10.10.2.221_node2] stopping ...
[2019-01-07T19:23:52,940][INFO][o.e.n.Node] [10.10.2.221_node2] stopped
[2019-01-07T19:23:52,941][INFO][o.e.n.Node] [10.10.2.221_node2] closing ...
[2019-01-07T19:23:52,954][INFO][o.e.n.Node] [10.10.2.221_node2] closed

```

the /etc/security/limits.conf content is as follow:

```auto
* soft nofile 65536
* hard nofile 131072
* soft nproc 2048
* hard nproc 4096
* soft memlock unlimited
* hard memlock unlimited

```

the memlock is already set unlimited and the [soft nofile] is 65536, but i don't known why this configuration didn't work

thanks  
iamazy

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [January 8, 2019, 9:08am UTC](https://discuss.elastic.co/t/why-does-es-node-auto-down-itself/163194/2 "2019-01-08T09:08:00Z")

</div>

It seems, that despite the settings being configured, they are not applied properly. Either you may need to relogin (not sure how you start Elasticsearch) or your operating system has another mechanism of setting these and may potentially overwrite or ignore your configuration.

See [https://www.elastic.co/guide/en/elasticsearch/reference/6.5/setup-configuration-memory.html#bootstrap-memory\_lock](https://www.elastic.co/guide/en/elasticsearch/reference/6.5/setup-configuration-memory.html#bootstrap-memory_lock) for more configuration options and [https://www.elastic.co/guide/en/elasticsearch/reference/6.5/file-descriptors.html](https://www.elastic.co/guide/en/elasticsearch/reference/6.5/file-descriptors.html) for more infos about file descriptors.

--Alex

---

<div class="post-metadata">

**Author:** ![iamazy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iamazy/32/36501_2.png) [@iamazy](https://discuss.elastic.co/u/iamazy)\
**Post date:** [January 8, 2019, 10:39am UTC](https://discuss.elastic.co/t/why-does-es-node-auto-down-itself/163194/3 "2019-01-08T10:39:05Z")

</div>

i will try it  
Thanks for the response  
iamazy

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 5, 2019, 10:39am UTC](https://discuss.elastic.co/t/why-does-es-node-auto-down-itself/163194/4 "2019-02-05T10:39:05Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
