# Nodes do not find the initial master node

**URL:** <https://discuss.elastic.co/t/nodes-do-not-find-the-initial-master-node/362405>\
**Category:** Elasticsearch\
**Tags:** docker\
**Created:** [July 2, 2024, 7:40pm UTC](https://discuss.elastic.co/t/nodes-do-not-find-the-initial-master-node/362405 "2024-07-02T19:40:13Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![jhonsouza](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jhonsouza/32/134261_2.png) [@jhonsouza](https://discuss.elastic.co/u/jhonsouza)\
**Post date:** [July 2, 2024, 7:40pm UTC](https://discuss.elastic.co/t/nodes-do-not-find-the-initial-master-node/362405/1 "2024-07-02T19:40:13Z")

</div>

Hi, guys!  
I'm trying to run the Elasticsearch on AWS ECS. I have a problem at the moment, the nodes try to join the cluster. When the other nodes are in the same instance that the initial master node, they can to join the cluster, but when the other nodes are in a different instance, they are got not join to the cluster. Below the elasticsearch.yml I use.

```auto
Initial master node
cluster:
  name: '${STACK_NAME}'
  initial_master_nodes: 
    - '${HOSTNAME}-es-initial-master'
  routing:
    allocation:
      awareness: 
        attributes: aws_availability_zone
node:
  name: '${HOSTNAME}-es-initial-master'
  roles: [master]
path: 
  data: /usr/share/elasticsearch/data
  logs: /usr/share/elasticsearch/logs
bootstrap:
  memory_lock: true
discovery:
  zen: 
    minimum_master_nodes: 2
  ec2:
    endpoint: ec2.us-east-2.amazonaws.com
network:
  host: 0.0.0.0
xpack:
  monitoring:
    collection:
      interval: 10s
      enabled: true
  security:
    enabled: true
cloud:
  node:
    auto_attributes: true
http:
  cors:
    enabled: true
    allow-origin: "*"
ingest:
  geoip:
    downloader:
      enabled: false

```

```auto
Data node
cluster:
  name: '${STACK_NAME}'
node:
  name: '${HOSTNAME}-es-data-${SUFFIX}'
  roles: ["data"]
path: 
  data: /usr/share/elasticsearch/data
  logs: /usr/share/elasticsearch/logs
bootstrap:
  memory_lock: true
discovery:
  seed_providers: ec2
  ec2:
    endpoint: ec2.us-east-2.amazonaws.com
  seed_hosts: []
s3:
  client:
    default:
      endpoint: s3.us-east-2.amazonaws.com
network:
  host: 0.0.0.0
xpack:
  monitoring:
    collection:
      interval: 10s
      enabled: true
  security:
    enabled: true
cloud:
  node:
    auto_attributes: true
http:
  cors:
    enabled: true
    allow-origin: "*"
ingest:
  geoip:
    downloader:
      enabled: false

```

If someone can help me, I appreciate a lot of!!

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [July 2, 2024, 9:01pm UTC](https://discuss.elastic.co/t/nodes-do-not-find-the-initial-master-node/362405/2 "2024-07-02T21:01:57Z")

</div>

See [these docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/discovery-troubleshooting.html) for guidance about how to troubleshoot discovery problems, including the things to look for in logs etc. If you need help understanding your logs, please share them here.

---

<div class="post-metadata">

**Author:** ![jhonsouza](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jhonsouza/32/134261_2.png) [@jhonsouza](https://discuss.elastic.co/u/jhonsouza)\
**Post date:** [July 3, 2024, 2:02pm UTC](https://discuss.elastic.co/t/nodes-do-not-find-the-initial-master-node/362405/3 "2024-07-03T14:02:58Z")

</div>

Hi David. Thanks for the docs, this will be very helpful. I found this looking at the logs. The node find the eligible master, but can't complete the connection.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/9/a92040330226318aa16982dadd3c8b4ca549a485.png)

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [July 3, 2024, 2:19pm UTC](https://discuss.elastic.co/t/nodes-do-not-find-the-initial-master-node/362405/4 "2024-07-03T14:19:26Z")

</div>

Yep that'd do it - [these docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html#modules-network-binding-publishing) are what you need here.

---

<div class="post-metadata">

**Author:** ![jhonsouza](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jhonsouza/32/134261_2.png) [@jhonsouza](https://discuss.elastic.co/u/jhonsouza)\
**Post date:** [July 3, 2024, 2:29pm UTC](https://discuss.elastic.co/t/nodes-do-not-find-the-initial-master-node/362405/5 "2024-07-03T14:29:35Z")

</div>

Thanks a lot, David! I'll read this now!

---

<div class="post-metadata">

**Author:** ![jhonsouza](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jhonsouza/32/134261_2.png) [@jhonsouza](https://discuss.elastic.co/u/jhonsouza)\
**Post date:** [July 3, 2024, 7:43pm UTC](https://discuss.elastic.co/t/nodes-do-not-find-the-initial-master-node/362405/6 "2024-07-03T19:43:17Z")

</div>

If I configure the `network.publish_host` and `network.bind_host` with the value: `0.0.0.0`. This is not should resolve my problem?

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [July 4, 2024, 6:33am UTC](https://discuss.elastic.co/t/nodes-do-not-find-the-initial-master-node/362405/7 "2024-07-04T06:33:11Z")

</div>

`0.0.0.0` can be fairly trappy for `network.publish_host`, especially if there's some kind of proxying or NAT going on as appears to be the case in your environment. I'd recommend being more specific. The log message you shared indicates that one possible step towards a resolution would be to specify `network.publish_host: 172.30.5.137` on node `deccdfd16c64-es-initial-master`.

---

<div class="post-metadata">

**Author:** ![jhonsouza](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jhonsouza/32/134261_2.png) [@jhonsouza](https://discuss.elastic.co/u/jhonsouza)\
**Post date:** [July 4, 2024, 1:39pm UTC](https://discuss.elastic.co/t/nodes-do-not-find-the-initial-master-node/362405/8 "2024-07-04T13:39:33Z")

</div>

Thanks, @DavidTurner!! I changed the `network.publish_host: 0.0.0.0` to `network.publish_host: _ec2_`. Now I receive a new error

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/0/d0a77e495365706b9dab4b0931220e8898e126e5.png)

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [July 5, 2024, 7:51am UTC](https://discuss.elastic.co/t/nodes-do-not-find-the-initial-master-node/362405/9 "2024-07-05T07:51:39Z")

</div>

Would you copy the text of the errors (formatted with the `</>` button) rather than screenshots? Screenshots are pretty much unreadable here. And include the whole message, not just the few lines you screenshotted, because there's important detailed missing here.

---

<div class="post-metadata">

**Author:** ![jhonsouza](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jhonsouza/32/134261_2.png) [@jhonsouza](https://discuss.elastic.co/u/jhonsouza)\
**Post date:** [July 5, 2024, 7:27pm UTC](https://discuss.elastic.co/t/nodes-do-not-find-the-initial-master-node/362405/10 "2024-07-05T19:27:03Z")

</div>

Sorry for delay, here are the logs

```auto
{"type": "server", "timestamp": "2024-07-05T18:48:15,080Z", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "dev-es", "node.name": "es-kibana", "message": "failed to join {es-initial-master}{0uGjtOVrQ2qUspiQo9tMLw}{zzUo1K6cRwmdJpkuzBNBAw}{172.30.4.30}{172.30.4.30:9300}{m}{aws_availability_zone=us-east-2b, xpack.installed=true, transform.node=false} with JoinRequest{sourceNode={es-kibana}{2Slv2ZZdRQa1yNV7NPazyw}{XsiGsq4-QVqaMl44QKp8ig}{172.17.0.8}{172.17.0.8:9300}{dir}{aws_availability_zone=us-east-2a, xpack.installed=true, transform.node=false}, minimumTerm=1, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={es-kibana}{2Slv2ZZdRQa1yNV7NPazyw}{XsiGsq4-QVqaMl44QKp8ig}{172.17.0.8}{172.17.0.8:9300}{dir}{aws_availability_zone=us-east-2a, xpack.installed=true, transform.node=false}, targetNode={es-initial-master}{0uGjtOVrQ2qUspiQo9tMLw}{zzUo1K6cRwmdJpkuzBNBAw}{172.30.4.30}{172.30.4.30:9300}{m}{aws_availability_zone=us-east-2b, xpack.installed=true, transform.node=false}}]}", 
"stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [es-initial-master][172.17.0.10:9300][internal:cluster/coordination/join]",
"Caused by: org.elasticsearch.transport.ConnectTransportException: [es-kibana][172.17.0.8:9300] handshake failed. unexpected remote node {es-client-18d27a6938af}{oxS_gycZTI6_T_pjkLeXTw}{WZicVUy9SeibYfn0xkuAqg}{172.17.0.8}{172.17.0.8:9300}{r}{aws_availability_zone=us-east-2b, xpack.installed=true, transform.node=false}",
"at org.elasticsearch.transport.TransportService.lambda$connectionValidator$6(TransportService.java:468) ~[elasticsearch-7.17.15.jar:7.17.15]",
"at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:95) ~[elasticsearch-7.17.15.jar:7.17.15]",
"at org.elasticsearch.transport.TransportService.lambda$handshake$9(TransportService.java:577) ~[elasticsearch-7.17.15.jar:7.17.15]",
"at org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:219) ~[elasticsearch-7.17.15.jar:7.17.15]",
"at org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:43) ~[elasticsearch-7.17.15.jar:7.17.15]",
"at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1471) ~[elasticsearch-7.17.15.jar:7.17.15]",
"at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1471) ~[elasticsearch-7.17.15.jar:7.17.15]",
"at org.elasticsearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:352) ~[elasticsearch-7.17.15.jar:7.17.15]",
"at org.elasticsearch.transport.InboundHandler.lambda$handleResponse$1(InboundHandler.java:340) ~[elasticsearch-7.17.15.jar:7.17.15]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) ~[elasticsearch-7.17.15.jar:7.17.15]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]",
"at java.lang.Thread.run(Thread.java:1583) [?:?]"] }

```

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [July 6, 2024, 3:59pm UTC](https://discuss.elastic.co/t/nodes-do-not-find-the-initial-master-node/362405/11 "2024-07-06T15:59:33Z")

</div>

> [@jhonsouza](#):
>
> `[es-kibana][172.17.0.8:9300] handshake failed. unexpected remote node {es-client-18d27a6938af}...{172.17.0.8:9300}`

Seems that you have two nodes that both claim to be at `172.17.0.8:9300`, that's not going to work. Every node needs its own address.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [July 8, 2024, 4:38am UTC](https://discuss.elastic.co/t/nodes-do-not-find-the-initial-master-node/362405/12 "2024-07-08T04:38:43Z")

</div>

I opened [Clarify logs/errors re. publish addresses by DaveCTurner · Pull Request #110570 · elastic/elasticsearch · GitHub](https://github.com/elastic/elasticsearch/pull/110570) to clarify this point, plus reword the log messages to be clearer and add links to the relevant docs.

---

<div class="post-metadata">

**Author:** ![jhonsouza](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jhonsouza/32/134261_2.png) [@jhonsouza](https://discuss.elastic.co/u/jhonsouza)\
**Post date:** [July 8, 2024, 8:54pm UTC](https://discuss.elastic.co/t/nodes-do-not-find-the-initial-master-node/362405/13 "2024-07-08T20:54:50Z")

</div>

Hi, @DavidTurner! Thank you so much for your helping in the troubleshooting. It's working now, I'll document here the solution to helping others there are these problems.
