# Cluster nodes didn't recognize each others (ES-7.7.1) **URL:** \ **Category:** Elasticsearch\ **Created:** [January 12, 2021, 2:39pm UTC](https://discuss.elastic.co/t/cluster-nodes-didnt-recognize-each-others-es-7-7-1/260839 "2021-01-12T14:39:32Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Guillaume\_Tiberi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guillaume_tiberi/32/82098_2.png) [@Guillaume\_Tiberi](https://discuss.elastic.co/u/Guillaume_Tiberi)\ **Post date:** [January 12, 2021, 2:39pm UTC](https://discuss.elastic.co/t/cluster-nodes-didnt-recognize-each-others-es-7-7-1/260839/1 "2021-01-12T14:39:33Z")
Hi I want to build a 5 nodes ES cluster but nodes didn't recognized each others. Could you help me to well config this cluster ? It's like node didn't recognize themself to be the master-node (node-3 looking for master node named node-3): ```auto [2021-01-12T14:55:48,544][WARN][o.e.c.c.ClusterFormationFailureHelper] [**node-3**] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node-1, node-2, **node-3**] to bootstrap a cluster ``` Is that because of searching in a different port (node in 9200, searching in 9300)? ```auto discovery will continue using [172.20.57.65:9300, 127.0.0.1:9300] from hosts providers ``` Why did he search for node at 9300 instead of 9200 ? elasticsearch nodes config: ```auto cluster.name: MYAPP_ES_7 node.name: node-1 # node-2 -3 -4 -5 for other nodes path.data: /elasticsearch/data/myapp_es7 path.logs: /SD5/people/myapp/elasticsearch/elasticsearch-7.7.1/logs bootstrap.memory_lock: false network.host: xxxxxxxxxxxxx http.port: 9200 # 9201 9202 9203 9204 for other nodes discovery.seed_hosts: ["xxxxxxxxxxxxx"] cluster.initial_master_nodes: ["node-1", "node-2", "node-3"] ``` Logs: ```auto [2021-01-12T14:55:48,544][WARN][o.e.c.c.ClusterFormationFailureHelper] [node-3] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node-1, node-2, node-3] to bootstrap a cluster: have discovered [{node-3}{AYBrgaSiQ0uZoQ0s4sJ6hw}{0ayIu41_QfGsPjFdRnB8Pw}{xxxxxxxxxxxxx}{172.20.57.65:9301}{dilmrt}{ml.machine_memory=211121090560, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [172.20.57.65:9300, 127.0.0.1:9300] from hosts providers and [{node-3}{AYBrgaSiQ0uZoQ0s4sJ6hw}{0ayIu41_QfGsPjFdRnB8Pw}{xxxxxxxxxxxxx}{172.20.57.65:9301}{dilmrt}{ml.machine_memory=211121090560, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0 [2021-01-12T14:55:48,566][WARN][o.e.c.c.ClusterFormationFailureHelper] [node-2] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node-1, node-2, node-3] to bootstrap a cluster: have discovered [{node-2}{dq9VCNiERJ-GjAaehzWoGQ}{HZ9kot15RBmubCOfG-B92w}{xxxxxxxxxxxxx}{172.20.57.65:9302}{dilmrt}{ml.machine_memory=211121090560, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [172.20.57.65:9300, 127.0.0.1:9300] from hosts providers and [{node-2}{dq9VCNiERJ-GjAaehzWoGQ}{HZ9kot15RBmubCOfG-B92w}{xxxxxxxxxxxxx}{172.20.57.65:9302}{dilmrt}{ml.machine_memory=211121090560, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0 [2021-01-12T14:55:48,731][WARN][o.e.c.c.ClusterFormationFailureHelper] [node-1] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node-1, node-2, node-3] to bootstrap a cluster: have discovered [{node-1}{W2m34wnPR3SiUZveBQkB_w}{740ABCRCR5KQ8c2FZVEtWg}{xxxxxxxxxxxxx}{172.20.57.65:9303}{dilmrt}{ml.machine_memory=211121090560, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [172.20.57.65:9300, 127.0.0.1:9300] from hosts providers and [{node-1}{W2m34wnPR3SiUZveBQkB_w}{740ABCRCR5KQ8c2FZVEtWg}{xxxxxxxxxxxxx}{172.20.57.65:9303}{dilmrt}{ml.machine_memory=211121090560, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0 [2021-01-12T14:55:49,262][WARN][o.e.c.c.ClusterFormationFailureHelper] [node-4] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node-1, node-2, node-3] to bootstrap a cluster: have discovered [{node-4}{nK7kqRc1SwWUkLLvtrrs-w}{Oss421FzT0qeoGSAGk7Vrw}{xxxxxxxxxxxxx}{172.20.57.65:9304}{dilmrt}{ml.machine_memory=211121090560, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [172.20.57.65:9300, 127.0.0.1:9300] from hosts providers and [{node-4}{nK7kqRc1SwWUkLLvtrrs-w}{Oss421FzT0qeoGSAGk7Vrw}{xxxxxxxxxxxxx}{172.20.57.65:9304}{dilmrt}{ml.machine_memory=211121090560, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0 [2021-01-12T14:55:49,487][WARN][o.e.c.c.ClusterFormationFailureHelper] [node-5] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node-1, node-2, node-3] to bootstrap a cluster: have discovered [{node-5}{KXvoyVvzTtmB_ZMZfv2g7A}{fQDxqin5QyOw1E6QyVrDhA}{xxxxxxxxxxxxx}{172.20.57.65:9305}{dilmrt}{ml.machine_memory=211121090560, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [172.20.57.65:9300, 127.0.0.1:9300] from hosts providers and [{node-5}{KXvoyVvzTtmB_ZMZfv2g7A}{fQDxqin5QyOw1E6QyVrDhA}{xxxxxxxxxxxxx}{172.20.57.65:9305}{dilmrt}{ml.machine_memory=211121090560, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0 ``` ---
**Author:** ![Imma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/imma/32/32013_2.png) [@Imma](https://discuss.elastic.co/u/Imma)\ **Post date:** [January 12, 2021, 3:45pm UTC](https://discuss.elastic.co/t/cluster-nodes-didnt-recognize-each-others-es-7-7-1/260839/2 "2021-01-12T15:45:37Z")
Hi Guillaume, Nodes will communicate internally using the transport protocol, which defaults to port 9300 if not specified. I'd suggest you explicitly specify the `transport.port` for each node. And then specifically declare it on the `cluster.initial_master_nodes` and `discovery.seed_hosts` if it's not the default `9300`. See: [https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html#discovery-settings](https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html#discovery-settings) Otherwise, discovery assumes the nodes are listening on port `9300` and it might not be the case. That is, if `node-1` binds the transport protocol to port `9300`, and `node-2` started on the same host, it would take `9301`, or the next `93xx` available port. If the instances run on different servers, they can actually all run on `http.port: 9200` and `transport.port 9300`. Though it's also a good practice to explicitly declare those anyway. Otherwise, if the port is taken, the nodes would start on the next available port, and they might not be discoverable. E.g. if you wanted to set different ports: **Node 1** ```auto node.name: node-1 http.port: 9200 transport.port: 9300 cluster.initial_master_nodes: ["node-1:9300", "node-2:9301", "node-3:9302"] discovery.seed_hosts: ["node-1:9300", "node-2:9301", "node-3:9302", "node-4:9303", "node-5:9304"] ``` **Node 2** ```auto node.name: node-2 http.port: 9201 transport.port: 9301 cluster.initial_master_nodes: ["node-1:9300", "node-2:9301", "node-3:9302"] discovery.seed_hosts: ["node-1:9300", "node-2:9301", "node-3:9302", "node-4:9303", "node-5:9304"] ``` You can have a look at the [discovery and cluster formation settings](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/modules-discovery-settings.html) and how the [cluster if formed](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/modules-discovery.html). ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [January 12, 2021, 3:53pm UTC](https://discuss.elastic.co/t/cluster-nodes-didnt-recognize-each-others-es-7-7-1/260839/3 "2021-01-12T15:53:13Z")
> [@Imma](#): > > `cluster.initial_master_nodes: ["node-1:9300", "node-2:9301", "node-3:9302"]` That's not right, `cluster.initial_master_nodes` should be a list of node names, not address/port pairs. The setting in the OP is correct. ---
**Author:** ![Guillaume\_Tiberi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guillaume_tiberi/32/82098_2.png) [@Guillaume\_Tiberi](https://discuss.elastic.co/u/Guillaume_Tiberi)\ **Post date:** [January 13, 2021, 11:24am UTC](https://discuss.elastic.co/t/cluster-nodes-didnt-recognize-each-others-es-7-7-1/260839/4 "2021-01-13T11:24:50Z")
@Imma Thank you for your answer. I try to change `transport.port` to an other port for all nodes, or by assigning one port for each node but this didn't fix my issue. @DavidTurner > [@DavidTurner](#): > > The setting in the OP is correct. Do you mean this config looks to be correct, which imply my problem could come from server configuration issue? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [January 13, 2021, 11:41am UTC](https://discuss.elastic.co/t/cluster-nodes-didnt-recognize-each-others-es-7-7-1/260839/5 "2021-01-13T11:41:07Z")
> [@Guillaume\_Tiberi](#): > > Do you mean this config looks to be correct, which imply my problem could come from server configuration issue? I only mean that this line in your config is correct: ```plaintext cluster.initial_master_nodes: ["node-1", "node-2", "node-3"] ``` The rest of Imma's post is good advice, you should follow it. ---
**Author:** ![Imma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/imma/32/32013_2.png) [@Imma](https://discuss.elastic.co/u/Imma)\ **Post date:** [January 13, 2021, 12:10pm UTC](https://discuss.elastic.co/t/cluster-nodes-didnt-recognize-each-others-es-7-7-1/260839/6 "2021-01-13T12:10:52Z")
What's the value of [`network.host`](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/modules-network.html#common-network-settings) in your configuration? If you are binding to local IPs like `192.168.0.1`, `192.168.0.2`, etc., you need to configure those on `discovery.seed_hosts`. That is if the node names `node-1`, `node-2`, etc. do not resolve to the right bound IP. If not specified, the setting defaults to `_local_` and instances will only bind to localhost, not being visible from others hosts. Assuming you are running the 5 instances on different hosts, of course. We see the special value [`_site_`](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html#network-interface-values) being used a lot, as it binds to any site-local addresses on the system. Though this will depend on your case. I hope this helps. ---
**Author:** ![Guillaume\_Tiberi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guillaume_tiberi/32/82098_2.png) [@Guillaume\_Tiberi](https://discuss.elastic.co/u/Guillaume_Tiberi)\ **Post date:** [January 15, 2021, 5:05pm UTC](https://discuss.elastic.co/t/cluster-nodes-didnt-recognize-each-others-es-7-7-1/260839/7 "2021-01-15T17:05:25Z")
@Imma I fix my problem using your solution of defining one `transport.port` for each node and `discovery.seed_hosts` with `host:transport.port` format (instead of `node.name:transport.port` as you did, look like a copy/paste typo). After that changes, I had to delete the `path.data` folder content in order to make the cluster build correctly. **Node 1** ```auto cluster.name: MYAPP_ES_7 node.name: node-1 network.host: xxxxxxxxxxxxx http.port: 9201 transport.port: 9301 discovery.seed_hosts: ["xxxxxxxxxxxxx:9301, xxxxxxxxxxxxx:9302, ..."] cluster.initial_master_nodes: ["node-1", "node-2", "node-3"] ``` **Node 2** ```auto cluster.name: MYAPP_ES_7 node.name: node-2 network.host: xxxxxxxxxxxxx http.port: 9201 transport.port: 9301 discovery.seed_hosts: ["xxxxxxxxxxxxx:9301, xxxxxxxxxxxxx:9302, ..."] cluster.initial_master_nodes: ["node-1", "node-2", "node-3"] ``` Thank you ! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 12, 2021, 5:05pm UTC](https://discuss.elastic.co/t/cluster-nodes-didnt-recognize-each-others-es-7-7-1/260839/8 "2021-02-12T17:05:33Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.