# ELK 5.4 data node is failing to join the cluster, it is stuck at discovery phase **URL:** \ **Category:** Elasticsearch\ **Created:** [May 9, 2018, 3:53pm UTC](https://discuss.elastic.co/t/elk-5-4-data-node-is-failing-to-join-the-cluster-it-is-stuck-at-discovery-phase/131197 "2018-05-09T15:53:03Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![chakri276](https://avatars.discourse-cdn.com/v4/letter/c/e99b99/32.png) [@chakri276](https://discuss.elastic.co/u/chakri276)\ **Post date:** [May 9, 2018, 3:53pm UTC](https://discuss.elastic.co/t/elk-5-4-data-node-is-failing-to-join-the-cluster-it-is-stuck-at-discovery-phase/131197/1 "2018-05-09T15:53:03Z")
I am using Elasticsearch 5.4, I have 3 master nodes and 8 data notes in which one data node stopped its elastic search process abruptly as its supervisord is down. I have 111 unassigned shrads and the reason for that is, **"can\_allocate" : "no\_valid\_shard\_copy",** \*\* "allocate\_explanation" : "cannot allocate because a previous copy of the primary shard existed but can no longer be found on the nodes in the cluster",\*\* I am thinking the node which stopped should have the primary shard, That node is failing to join the cluster, from the logs I see below, [2018-05-09T11:39:55,759][INFO][o.e.n.Node] [ah-1007168-003] initializing ... [2018-05-09T11:39:55,812][INFO][o.e.e.NodeEnvironment] [ah-1007168-003] using [1] data paths, mounts [[/apps/cie\_dashboard (/dev/mapper/volgrp02-appscie\_dashboard)]], net usable\_space [529.1gb], net total\_space [899.5gb], spins? [possibly], types [xfs] [2018-05-09T11:39:55,813][INFO][o.e.e.NodeEnvironment] [ah-1007168-003] heap size [23.9gb], compressed ordinary object pointers [true] [2018-05-09T11:39:56,141][INFO][o.e.n.Node] [ah-1007168-003] node name [ah-1007168-003], node ID [iUAPJ0glTV2fP5\_isJE4Jg] [2018-05-09T11:39:56,141][INFO][o.e.n.Node] [ah-1007168-003] version[5.4.0], pid[6077], build[780f8c4/2017-04-28T17:43:27.229Z], OS[Linux/3.10.0-693.17.1.el7.x86\_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0\_72/25.72-b15] [2018-05-09T11:39:56,914][INFO][o.e.p.PluginsService] [ah-1007168-003] loaded module [aggs-matrix-stats] [2018-05-09T11:39:56,914][INFO][o.e.p.PluginsService] [ah-1007168-003] loaded module [ingest-common] [2018-05-09T11:39:56,915][INFO][o.e.p.PluginsService] [ah-1007168-003] loaded module [lang-expression] [2018-05-09T11:39:56,915][INFO][o.e.p.PluginsService] [ah-1007168-003] loaded module [lang-groovy] [2018-05-09T11:39:56,915][INFO][o.e.p.PluginsService] [ah-1007168-003] loaded module [lang-mustache] [2018-05-09T11:39:56,915][INFO][o.e.p.PluginsService] [ah-1007168-003] loaded module [lang-painless] [2018-05-09T11:39:56,915][INFO][o.e.p.PluginsService] [ah-1007168-003] loaded module [percolator] [2018-05-09T11:39:56,915][INFO][o.e.p.PluginsService] [ah-1007168-003] loaded module [reindex] [2018-05-09T11:39:56,915][INFO][o.e.p.PluginsService] [ah-1007168-003] loaded module [transport-netty3] [2018-05-09T11:39:56,915][INFO][o.e.p.PluginsService] [ah-1007168-003] loaded module [transport-netty4] [2018-05-09T11:39:56,916][INFO][o.e.p.PluginsService] [ah-1007168-003] no plugins loaded [2018-05-09T11:39:58,062][INFO][o.e.d.DiscoveryModule] [ah-1007168-003] using discovery type [zen] [2018-05-09T11:39:59,561][INFO][o.e.n.Node] [ah-1007168-003] initialized [2018-05-09T11:39:59,561][INFO][o.e.n.Node] [ah-1007168-003] starting ... [2018-05-09T11:39:59,677][INFO][o.e.t.TransportService] [ah-1007168-003] publish\_address {171.135.145.72:9300}, bound\_addresses {0.0.0.0:9300} [2018-05-09T11:39:59,683][INFO][o.e.b.BootstrapChecks] [ah-1007168-003] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks [2018-05-09T11:39:59,687][ERROR][o.e.b.Bootstrap] [ah-1007168-003] node validation exception bootstrap checks failed max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536] [2018-05-09T11:39:59,690][INFO][o.e.n.Node] [ah-1007168-003] stopping ... [2018-05-09T11:39:59,713][INFO][o.e.n.Node] [ah-1007168-003] stopped [2018-05-09T11:39:59,713][INFO][o.e.n.Node] [ah-1007168-003] closing ... [2018-05-09T11:39:59,721][INFO][o.e.n.Node] [ah-1007168-003] closed My /etc/security/limits.conf values are already set to, root - nofile 100000 ahselkpd - nofile 82919 ahselkpd - memlock unlimited # #unlimited memory lock for elk\_user my ulimit -n for ahselkpd is 82919 my ulimit -n for root is 100000 Also the value for vm.max\_map\_count=262144 is set globally under /etc/sysctl.conf Please help how do I resolve this issue? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 6, 2018, 3:53pm UTC](https://discuss.elastic.co/t/elk-5-4-data-node-is-failing-to-join-the-cluster-it-is-stuck-at-discovery-phase/131197/2 "2018-06-06T15:53:07Z")
Closed on June 6, 2018, 3:53pm UTC This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.