# Not able to join a node in cluster

**URL:** <https://discuss.elastic.co/t/not-able-to-join-a-node-in-cluster/106966>\
**Category:** Elasticsearch\
**Created:** [November 9, 2017, 5:37am UTC](https://discuss.elastic.co/t/not-able-to-join-a-node-in-cluster/106966 "2017-11-09T05:37:51Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![rangitgaddam](https://avatars.discourse-cdn.com/v4/letter/r/34f0e0/32.png) [@rangitgaddam](https://discuss.elastic.co/u/rangitgaddam)\
**Post date:** [November 9, 2017, 5:37am UTC](https://discuss.elastic.co/t/not-able-to-join-a-node-in-cluster/106966/1 "2017-11-09T05:37:51Z")

</div>

Hi i am not able to join simple data node(running in another server) to existing cluster(master running in another server) with in a network please check my config files below

**elasticserach.yml(for master node)**

[cluster.name](http://cluster.name): kafka-connector  
[node.name](http://node.name): node-1  
node.master : true  
node.data : false  
path.data: /home/user/Kafka-ELK/KafkaElkConnector/data/elasticsearch  
path.logs: /home/user/Kafka-ELK/KafkaElkConnector/data/elasticsearch-logs  
bootstrap.memory\_lock: true  
network.host: "10.0.0.103"  
http.port: 9200  
transport.host: localhost  
transport.tcp.port: 9300  
discovery.zen.ping.unicast.hosts: ["10.0.0.103","10.0.0.114"]

**elasticsearch.yml(in another server for data node)**

[cluster.name](http://cluster.name): kafka-connector

[node.name](http://node.name): node-2  
node.master : false  
node.data : true

path.data: /home/user/Kafka-ELK/KafkaElkConnector/data/elasticsearch

path.logs: /home/user/Kafka-ELK/KafkaElkConnector/data/elasticsearch-logs  
bootstrap.memory\_lock: true  
network.host: "10.0.0.114"  
http.port: 9201

transport.host: localhost  
transport.tcp.port: 9301  
discovery.zen.ping.unicast.hosts: ["10.0.0.103"]

My cluster health is not showing data nodes

{  
"cluster\_name" : "kafka-connector",  
"status" : "red",  
"timed\_out" : false,  
"number\_of\_nodes" : 1,  
"number\_of\_data\_nodes" : 0,  
"active\_primary\_shards" : 0,  
"active\_shards" : 0,  
"relocating\_shards" : 0,  
"initializing\_shards" : 0,  
"unassigned\_shards" : 12,  
"delayed\_unassigned\_shards" : 0,  
"number\_of\_pending\_tasks" : 0,  
"number\_of\_in\_flight\_fetch" : 0,  
"task\_max\_waiting\_in\_queue\_millis" : 0,  
"active\_shards\_percent\_as\_number" : 0.0  
}

---

<div class="post-metadata">

**Author:** ![zqc0512](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zqc0512/32/32141_2.png) [@zqc0512](https://discuss.elastic.co/u/zqc0512)\
**Post date:** [November 9, 2017, 6:05am UTC](https://discuss.elastic.co/t/not-able-to-join-a-node-in-cluster/106966/2 "2017-11-09T06:05:27Z")

</div>

discovery.zen.ping.unicast.hosts: ["10.0.0.103","10.0.0.114"]  
this is need add in node-2.  
and transport.host: localhost need not use localhost.

---

<div class="post-metadata">

**Author:** ![rangitgaddam](https://avatars.discourse-cdn.com/v4/letter/r/34f0e0/32.png) [@rangitgaddam](https://discuss.elastic.co/u/rangitgaddam)\
**Post date:** [November 9, 2017, 6:08am UTC](https://discuss.elastic.co/t/not-able-to-join-a-node-in-cluster/106966/3 "2017-11-09T06:08:48Z")

</div>

> [@zqc0512](#):
>
> "10.0.0.103","10.0.0.114"

Do you mean that transport.host property is no more need?

How come that will be possible without transport.host and the node is getting closed automatically if i start the node with other than localhost in transport.host please check out the logs below

[2017-11-09T11:44:37,142][INFO][o.e.b.BootstrapChecks] [node-1] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks  
**[2017-11-09T11:44:37,144][ERROR][o.e.b.Bootstrap] [node-1] node validation exception**  
**[3] bootstrap checks failed**  
[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]  
[2]: memory locking requested for elasticsearch process but memory is not locked  
[3]: max virtual memory areas vm.max\_map\_count [65530] is too low, increase to at least [262144]  
[2017-11-09T11:44:37,145][INFO][o.e.n.Node] [node-1] stopping ...  
[2017-11-09T11:44:37,159][INFO][o.e.n.Node] [node-1] stopped  
[2017-11-09T11:44:37,159][INFO][o.e.n.Node] [node-1] closing ...  
[2017-11-09T11:44:37,165][INFO][o.e.n.Node] [node-1] closed

---

<div class="post-metadata">

**Author:** ![zqc0512](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zqc0512/32/32141_2.png) [@zqc0512](https://discuss.elastic.co/u/zqc0512)\
**Post date:** [November 9, 2017, 6:23am UTC](https://discuss.elastic.co/t/not-able-to-join-a-node-in-cluster/106966/4 "2017-11-09T06:23:07Z")

</div>

**[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]**  
**[2]: memory locking requested for elasticsearch process but memory is not locked**  
**[3]: max virtual memory areas vm.max\_map\_count [65530] is too low, increase to at least [262144]**  
this output 3 error,  
you need change it in system...

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [November 9, 2017, 6:27am UTC](https://discuss.elastic.co/t/not-able-to-join-a-node-in-cluster/106966/5 "2017-11-09T06:27:33Z")

</div>

You have different problems:

If you change `transport.tcp.port: 9301`, then you need to change `discovery.zen.ping.unicast.hosts: ["10.0.0.103","10.0.0.114:9301"]`.

But keep the maximum default values as possible. Here I'd simplify the configuration with:

Node 1:

```auto
cluster.name: kafka-connector
node.name: node-1
node.master : true
node.data : false
path.data: /home/user/Kafka-ELK/KafkaElkConnector/data/elasticsearch
path.logs: /home/user/Kafka-ELK/KafkaElkConnector/data/elasticsearch-logs
bootstrap.memory_lock: true
network.host: "10.0.0.103"
discovery.zen.ping.unicast.hosts: ["10.0.0.103"]

```

And Node2:

```auto
cluster.name: kafka-connector
node.name: node-2
node.master : false
node.data : true
path.data: /home/user/Kafka-ELK/KafkaElkConnector/data/elasticsearch
path.logs: /home/user/Kafka-ELK/KafkaElkConnector/data/elasticsearch-logs
bootstrap.memory_lock: true
network.host: "10.0.0.114"
discovery.zen.ping.unicast.hosts: ["10.0.0.103"]

```

Note that the data only node does not have to be declared in `discovery.zen.ping.unicast.hosts`. Only master eligible nodes needs to be declared there.

Then the last problem is bootstrap.

Please read [https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html)

---

<div class="post-metadata">

**Author:** ![rangitgaddam](https://avatars.discourse-cdn.com/v4/letter/r/34f0e0/32.png) [@rangitgaddam](https://discuss.elastic.co/u/rangitgaddam)\
**Post date:** [November 9, 2017, 12:20pm UTC](https://discuss.elastic.co/t/not-able-to-join-a-node-in-cluster/106966/9 "2017-11-09T12:20:32Z")

</div>

@dadoonet @zqc0512 OOPS after so much time of debugging i made it to start working the Cluster and able to add the nodes.

---

<div class="post-metadata">

**Author:** ![zqc0512](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zqc0512/32/32141_2.png) [@zqc0512](https://discuss.elastic.co/u/zqc0512)\
**Post date:** [November 10, 2017, 1:45am UTC](https://discuss.elastic.co/t/not-able-to-join-a-node-in-cluster/106966/10 "2017-11-10T01:45:43Z")

</div>

> [@rangitgaddam](#):
>
> [1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
> 
> [2]: memory locking requested for elasticsearch process but memory is not locked
> 
> [3]: max virtual memory areas vm.max\_map\_count [65530] is too low, increase to at least [262144]

**see the three error. this is configure by system with /etc/security/limits.conf and use systcl cmd to change it**

---

<div class="post-metadata">

**Author:** ![dzyubanv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dzyubanv/32/24110_2.png) [@dzyubanv](https://discuss.elastic.co/u/dzyubanv)\
**Post date:** [November 17, 2017, 3:45pm UTC](https://discuss.elastic.co/t/not-able-to-join-a-node-in-cluster/106966/11 "2017-11-17T15:45:46Z")

</div>

Hi Ranjith,  
I'm experiencing the same issue you resolved alredy.  
Just started to use es recently on two RHEL servers.  
Could you please share your two final yml files and what else did you, may be on the system level, to make your two nodes cluster work properly.  
Appreciate your help.

---

<div class="post-metadata">

**Author:** ![rangitgaddam](https://avatars.discourse-cdn.com/v4/letter/r/34f0e0/32.png) [@rangitgaddam](https://discuss.elastic.co/u/rangitgaddam)\
**Post date:** [November 20, 2017, 11:01am UTC](https://discuss.elastic.co/t/not-able-to-join-a-node-in-cluster/106966/12 "2017-11-20T11:01:28Z")

</div>

What is the issue you are facing so that will be clear to me.

---

<div class="post-metadata">

**Author:** ![dzyubanv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dzyubanv/32/24110_2.png) [@dzyubanv](https://discuss.elastic.co/u/dzyubanv)\
**Post date:** [November 20, 2017, 12:16pm UTC](https://discuss.elastic.co/t/not-able-to-join-a-node-in-cluster/106966/13 "2017-11-20T12:16:46Z")

</div>

Im running two nodes cluster on separate RHEL servers. One node is with master and data roles, second one is with data role.Nodes started but are not joining the cluster. When I commented out the "transport.host: localhost" then starting first node failed and es log file shows that it started and then failed to pass the bootstrap process:  
-max file des riptor 4096 increase to 65536  
-max virtual .emory vm.max\_map\_count 65530 increase to 262144  
-system call filters failed to install, fix or disable it  
Could you tell me please:  
\*what exactly did you do to pass the es bootstrap process and disappear these 3 messages ?  
\*or may be you changed something else ?  
\*my 2 yml files now are like yours initial yml files, but to my understanding you changed them too, so what are your modified yml files, allowing es cluster work ok ?  
\*which operating system your es instances are running on ?  
Thanks in advance

---

<div class="post-metadata">

**Author:** ![rangitgaddam](https://avatars.discourse-cdn.com/v4/letter/r/34f0e0/32.png) [@rangitgaddam](https://discuss.elastic.co/u/rangitgaddam)\
**Post date:** [November 20, 2017, 1:12pm UTC](https://discuss.elastic.co/t/not-able-to-join-a-node-in-cluster/106966/14 "2017-11-20T13:12:58Z")

</div>

Add bootstrap.system\_call\_filter: false to your both yml files and give only the master node host in the property of **discovery.zen.ping.unicast.hosts** in both the files and do not the change the default ports of _ **\_transport.port** _ let it be 9300 and transport. host must point to your system IP address of your system (for example 10.0.0.101)

increase the limits by running the following command as root  
**sysctl -w vm.max\_map\_count=262144**

this is configure by system with /etc/security/limits.conf and use systcl cmd to change it and add the below lines at the end of the file

\*\*\* soft nofile 64000\*\*  
\*\*\* hard nofile 64000\*\*  
\*\*\* soft nproc 63000\*\*  
\*\*\* hard nproc 63000\*\*

**yourusername soft nofile 65536**  
**yourusername hard nofile 65536**  
\*\* noproc 2048\*\*

Make sure to start elasticsearch node by the username given in /etc/security/limits.conf

exit the CLI sesion and relogin if you are accessing virtual machine otherwise reboot the system for applying changes.

Thanks.

---

<div class="post-metadata">

**Author:** ![dzyubanv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dzyubanv/32/24110_2.png) [@dzyubanv](https://discuss.elastic.co/u/dzyubanv)\
**Post date:** [November 20, 2017, 11:42pm UTC](https://discuss.elastic.co/t/not-able-to-join-a-node-in-cluster/106966/15 "2017-11-20T23:42:31Z")

</div>

Thanks for the response and help Ranjith.  
Your guidelines are very valuable.  
Will apply them and test es multi nodes cluster.

---

<div class="post-metadata">

**Author:** ![zqc0512](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zqc0512/32/32141_2.png) [@zqc0512](https://discuss.elastic.co/u/zqc0512)\
**Post date:** [November 21, 2017, 6:18am UTC](https://discuss.elastic.co/t/not-able-to-join-a-node-in-cluster/106966/16 "2017-11-21T06:18:53Z")

</div>

change in /etc/security/limits.conf need reboot machine. as i know.

---

<div class="post-metadata">

**Author:** ![dzyubanv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dzyubanv/32/24110_2.png) [@dzyubanv](https://discuss.elastic.co/u/dzyubanv)\
**Post date:** [November 21, 2017, 11:56am UTC](https://discuss.elastic.co/t/not-able-to-join-a-node-in-cluster/106966/17 "2017-11-21T11:56:05Z")

</div>

Will do it. Thanks.

---

<div class="post-metadata">

**Author:** ![dzyubanv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dzyubanv/32/24110_2.png) [@dzyubanv](https://discuss.elastic.co/u/dzyubanv)\
**Post date:** [November 27, 2017, 9:54pm UTC](https://discuss.elastic.co/t/not-able-to-join-a-node-in-cluster/106966/18 "2017-11-27T21:54:55Z")

</div>

Following your advises guys, I'm able run two nodes cluster on separate rhel server. The only one thing I'm not sure about is the bootstrap.memory\_lock: true in elasticsearch.yml files. If I commented it out at all then my multinode cluster works well. If I uncomment it then the node is not passing the bootstrap check validation. The ERROR is bootstrap checks failed [1]:memory locking requested for elasticsesrch provess but memory is not locked...stopped...closed.  
Should I comment the bootstrap.memory\_lock: true" out in prod or it is needed and something else need to be done on system level?  
Appreciate your advise.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 25, 2017, 9:55pm UTC](https://discuss.elastic.co/t/not-able-to-join-a-node-in-cluster/106966/19 "2017-12-25T21:55:18Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.