Elasticsearch (Version 7.9.0) cannot add new elasticsearch nodes to join an existing cluster

Elastic Stack Elasticsearch
1

Hello everyone,

I'm working with an Elasticsearch cluster that currently has 6 nodes. I'm trying to expand it to 9 nodes by adding 3 more nodes. However, I've only been able to successfully add one of the new nodes. The remaining two nodes are unable to join the cluster due to a "Not able to discover master-eligible node" error. Please help me:<

[2024-10-17T15:13:59,768][TRACE][o.e.d.PeerFinder         ] [node-7] probing master nodes from cluster state: nodes:
   {node-7}{Hfkqv0bzQWOmfHYxi9WMgA}{ePpdAXHnT8OpGmDZDWUogw}{10.36.13.70}{10.36.13.70:9300}{dilrt}{ml.machine_memory=33402966016, xpack.installed=true, data=warm, transform.node=true, ml.max_open_jobs=20}, local

[2024-10-17T15:13:59,769][TRACE][o.e.d.SeedHostsResolver  ] [node-7] resolved host [datalake01] to [10.36.13.201:9300]
[2024-10-17T15:13:59,769][TRACE][o.e.d.SeedHostsResolver  ] [node-7] resolved host [datalake02] to [10.36.13.202:9300]
[2024-10-17T15:13:59,769][TRACE][o.e.d.SeedHostsResolver  ] [node-7] resolved host [datalake03] to [10.36.13.203:9300]
[2024-10-17T15:13:59,769][TRACE][o.e.d.SeedHostsResolver  ] [node-7] resolved host [datalake04] to [10.36.13.204:9300]
[2024-10-17T15:13:59,769][TRACE][o.e.d.SeedHostsResolver  ] [node-7] resolved host [datalake05] to [10.36.13.214:9300]
[2024-10-17T15:13:59,769][TRACE][o.e.d.SeedHostsResolver  ] [node-7] resolved host [datalake06] to [10.36.13.217:9300]
[2024-10-17T15:13:59,770][TRACE][o.e.d.PeerFinder         ] [node-7] probing resolved transport addresses [10.36.13.201:9300, 10.36.13.202:9300, 10.36.13.203:9300, 10.36.13.204:9300, 10.36.13.214:9300, 10.36.13.217:9300]
2

I have something. after I enabled trace log mode, I noticed the handshake timed out even though the connection between the two servers was fine

[2024-10-21T16:53:03,711][DEBUG][o.e.d.PeerFinder         ] [node-10] Peer{transportAddress=10.36.13.217:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
org.elasticsearch.transport.ConnectTransportException: [][10.36.13.217:9300] handshake_timeout[10s]
        at org.elasticsearch.transport.TransportHandshaker.lambda$sendHandshake$1(TransportHandshaker.java:73) ~[elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:651) ~[elasticsearch-7.9.0.jar:7.9.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]