# ES node find master, but can't join the cluster

**URL:** <https://discuss.elastic.co/t/es-node-find-master-but-cant-join-the-cluster/256591>\
**Category:** Elasticsearch\
**Created:** [November 25, 2020, 3:20am UTC](https://discuss.elastic.co/t/es-node-find-master-but-cant-join-the-cluster/256591 "2020-11-25T03:20:17Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Todd](https://avatars.discourse-cdn.com/v4/letter/t/fbc32d/32.png) [@Todd](https://discuss.elastic.co/u/Todd)\
**Post date:** [November 25, 2020, 3:20am UTC](https://discuss.elastic.co/t/es-node-find-master-but-cant-join-the-cluster/256591/1 "2020-11-25T03:20:18Z")

</div>

Hi,  
These days I have found such a situation: one data node of the cluster has discoveried three master nodes, but it can't join the cluster. However, when I restart it, the data node joins successful !!!

#### ES version:

7.5.1

#### elasticsearch.yml

```auto
bootstrap.memory_lock: true
cluster.max_shards_per_node: 10000
cluster.name: billions-uat7.5.1
cluster.routing.allocation.allow_rebalance: always
cluster.routing.allocation.cluster_concurrent_rebalance: 5
cluster.routing.allocation.node_concurrent_recoveries: 10
cluster.routing.allocation.node_initial_primaries_recoveries: 20
cluster.routing.allocation.same_shard.host: true
discovery.seed_hosts:
- hw-sh-t-opslog-01:9310
- hw-sh-t-opslog-02:9310
- hw-sh-t-opslog-03:9310
http.port: 9201
indices.recovery.max_bytes_per_sec: 500mb
network.host: 0.0.0.0
node.attr.box_type: stale
node.data: true
node.master: false
transport.tcp.port: 9301
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/datanode_stale/billions-certificates.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/datanode_stale/billions-certificates.p12
xpack.security.transport.ssl.verification_mode: certificate

node.name: hw-sh-t-opslog-09-datanode_stale

path.data: /mnt/storage01/hw-sh-t-opslog-09-datanode_stale
path.logs: /mnt/storage01/elasticsearch/log/hw-sh-t-opslog-09-datanode_stale

action.auto_create_index: true
xpack.security.enabled: true

```

#### error log:

[https://github.com/cjx-great/issues/tree/main/es-7.5.1/master\_not\_discovered](https://github.com/cjx-great/issues/tree/main/es-7.5.1/master_not_discovered)

#### exporter monitor

![image](https://us1.discourse-cdn.com/elastic/original/3X/8/f/8fc40a5db16f83eaf74d9a95794c1b004eb8c769.png)

I want to know what caused this, and how to solve it ！

thanks 😄

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [November 25, 2020, 8:06am UTC](https://discuss.elastic.co/t/es-node-find-master-but-cant-join-the-cluster/256591/2 "2020-11-25T08:06:38Z")

</div>

In fact the node is joining the cluster fine, but then leaving again shortly afterwards. This message tells us why:

`node-left[{hw-sh-t-opslog-09-datanode_s tale}{b3mQRikmS1a4mXmyHKXu8g}{M3gSzZE6QnGFuQc1v_lEYQ}{10.221.46.135}{10.221.46.135:9301}{dil}{ml.machine_memory=67560857600, ml.m ax_open_jobs=20, xpack.installed=true, box_type=stale} reason: disconnected], term: 680, version: 991417, delta: removed {{hw-sh- t-opslog-09-datanode_stale}{b3mQRikmS1a4mXmyHKXu8g}{M3gSzZE6QnGFuQc1v_lEYQ}{10.221.46.135}{10.221.46.135:9301}{dil}{ml.machine_me mory=67560857600, ml.max_open_jobs=20, xpack.installed=true, box_type=stale}}`

The `reason: disconnected` bit indicates that something outside Elasticsearch was disrupting the TCP connections from the master to the problematic node.

---

<div class="post-metadata">

**Author:** ![Todd](https://avatars.discourse-cdn.com/v4/letter/t/fbc32d/32.png) [@Todd](https://discuss.elastic.co/u/Todd)\
**Post date:** [November 25, 2020, 8:34am UTC](https://discuss.elastic.co/t/es-node-find-master-but-cant-join-the-cluster/256591/3 "2020-11-25T08:34:54Z")

</div>

but when we restart it, the data node joins successful

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [November 25, 2020, 10:05am UTC](https://discuss.elastic.co/t/es-node-find-master-but-cant-join-the-cluster/256591/4 "2020-11-25T10:05:18Z")

</div>

Right, but it'll likely still get disconnected. That's what you are seeing?

It's likely some firewall somewhere that isn't allowing long lived connections.

---

<div class="post-metadata">

**Author:** ![Todd](https://avatars.discourse-cdn.com/v4/letter/t/fbc32d/32.png) [@Todd](https://discuss.elastic.co/u/Todd)\
**Post date:** [November 25, 2020, 10:37am UTC](https://discuss.elastic.co/t/es-node-find-master-but-cant-join-the-cluster/256591/5 "2020-11-25T10:37:04Z")

</div>

👌 I'll go to check the network side monitoring

thanks a lot

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 23, 2020, 10:37am UTC](https://discuss.elastic.co/t/es-node-find-master-but-cant-join-the-cluster/256591/6 "2020-12-23T10:37:20Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.