The cluster cannot discover the master or other nodes

Elastic Stack Elasticsearch
1

According to the prompt "this node must discover master-eligible nodes [es-node01, es-node02, es-node03]", es-node03 does not find other nodes. According to the results I searched, cluster.initial_master_nodes needs to be configured, but I have, then does this problem represent a network failure?

By the way, what does "node term 0, last-accepted version 0 in term 0" mean?

cluster.name: es

node.name: es-node03
node.attr.data: hot_warm
node.attr.size: 2T
node.attr.location: ydy_cluster
node.roles: [ master, data_hot, data_warm, data_content, data, ml, transform, remote_cluster_client, ingest ]

path.data: /home/elasticsearch/esdata/
path.logs: /var/log/elasticsearch

network.host: 0.0.0.0
http.port: 9200

discovery.seed_hosts: ["10.120.0.30:9200","10.120.0.14:9200","10.120.0.10:9200","10.120.0.8:9200","10.120.0.38:9200"]
cluster.initial_master_nodes: ["es-node01","es-node02","es-node03"]

#gateway.expected_nodes: 3
#gateway.recover_after_time: 20m
#gateway.recover_after_data_nodes: 3


xpack.security.enabled: false
xpack.monitoring.enabled: true
xpack.monitoring.collection.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate 
xpack.security.transport.ssl.keystore.path: elastic-stack-ca.p12
xpack.security.transport.ssl.truststore.path: elastic-stack-ca.p12

[2022-05-31T13:21:50,534][WARN ][o.e.c.c.ClusterFormationFailureHelper] [es-node03] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [es-node01, es-node02, es-node03] to bootstrap a cluster: have discovered [{es-node03}{s4abPK_pQzSnExqpykvvSg}{bdHnoop8R2KI9DKMneNGCA}{10.120.0.10}{10.120.0.10:9300}{dhilmrstw}]; discovery will continue using [10.120.0.30:9200, 10.120.0.14:9200, 10.120.0.10:9200, 10.120.0.8:9200, 10.120.0.38:9200] from hosts providers and [{es-node03}{s4abPK_pQzSnExqpykvvSg}{bdHnoop8R2KI9DKMneNGCA}{10.120.0.10}{10.120.0.10:9300}{dhilmrstw}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2022-05-31T13:22:00,534][WARN ][o.e.c.c.ClusterFormationFailureHelper] [es-node03] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [es-node01, es-node02, es-node03] to bootstrap a cluster: have discovered [{es-node03}{s4abPK_pQzSnExqpykvvSg}{bdHnoop8R2KI9DKMneNGCA}{10.120.0.10}{10.120.0.10:9300}{dhilmrstw}]; discovery will continue using [10.120.0.30:9200, 10.120.0.14:9200, 10.120.0.10:9200, 10.120.0.8:9200, 10.120.0.38:9200] from hosts providers and [{es-node03}{s4abPK_pQzSnExqpykvvSg}{bdHnoop8R2KI9DKMneNGCA}{10.120.0.10}{10.120.0.10:9300}{dhilmrstw}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2022-05-31T13:22:10,535][WARN ][o.e.c.c.ClusterFormationFailureHelper] [es-node03] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [es-node01, es-node02, es-node03] to bootstrap a cluster: have discovered [{es-node03}{s4abPK_pQzSnExqpykvvSg}{bdHnoop8R2KI9DKMneNGCA}{10.120.0.10}{10.120.0.10:9300}{dhilmrstw}]; discovery will continue using [10.120.0.30:9200, 10.120.0.14:9200, 10.120.0.10:9200, 10.120.0.8:9200, 10.120.0.38:9200] from hosts providers and [{es-node03}{s4abPK_pQzSnExqpykvvSg}{bdHnoop8R2KI9DKMneNGCA}{10.120.0.10}{10.120.0.10:9300}{dhilmrstw}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2022-05-31T13:22:11,563][WARN ][o.e.d.PeerFinder         ] [es-node03] address [10.120.0.10:9200], node [null], requesting [false] connection failed: [][10.120.0.10:9200] handshake_timeout[30s]
[2022-05-31T13:22:11,563][WARN ][o.e.d.PeerFinder         ] [es-node03] address [10.120.0.30:9200], node [null], requesting [false] connection failed: [][10.120.0.30:9200] handshake_timeout[30s]
[2022-05-31T13:22:11,564][WARN ][o.e.d.PeerFinder         ] [es-node03] address [10.120.0.38:9200], node [null], requesting [false] connection failed: [][10.120.0.38:9200] handshake_timeout[30s]
[2022-05-31T13:22:11,564][WARN ][o.e.d.PeerFinder         ] [es-node03] address [10.120.0.8:9200], node [null], requesting [false] connection failed: [][10.120.0.8:9200] handshake_timeout[30s]
[2022-05-31T13:22:12,563][WARN ][o.e.d.PeerFinder         ] [es-node03] address [10.120.0.14:9200], node [null], requesting [false] connection failed: [][10.120.0.14:9200] handshake_timeout[30s]

full log

2

Elasticsearch does not use HTTP or port 9200 to communicate between nodes, so remove the port number from the seed hosts. Port 9300 is used for communication within the cluster so make sure this is open.

2 Likes
3

thanks.

4 
[2022-05-31T15:58:30,020][WARN ][o.e.d.PeerFinder         ] [es-node05] address [10.120.0.8:9300], node [null], requesting [false] connection failed: [es-node04][10.120.0.8:9300] non-master-eligible node found
[2022-05-31T15:58:30,952][WARN ][o.e.c.c.ClusterFormationFailureHelper] [es-node05] master not discovered yet: have discovered [{es-node05}{bwiqzVz8TqOSvPzEQ-pVgA}{MNElIl_dQsGt6SEYv3qv6g}{10.120.0.38}{10.120.0.38:9300}{cdrs}]; discovery will continue using [10.120.0.7:9300, 10.120.0.24:9300, 10.120.0.31:9300, 10.120.0.8:9300, 10.120.0.22:9300] from hosts providers and [] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2022-05-31T15:58:31,020][WARN ][o.e.d.PeerFinder         ] [es-node05] address [10.120.0.8:9300], node [null], requesting [false] connection failed: [es-node04][10.120.0.8:9300] non-master-eligible node found
[2022-05-31T15:58:32,020][WARN ][o.e.d.PeerFinder         ] [es-node05] address [10.120.0.8:9300], node [null], requesting [false] connection failed: [es-node04][10.120.0.8:9300] non-master-eligible node found
[2022-05-31T15:58:33,002][WARN ][o.e.d.PeerFinder         ] [es-node05] address [10.120.0.7:9300], node [null], requesting [false] connection failed: [][10.120.0.7:9300] connect_exception: No route to host: /10.120.0.7:9300: No route to host
[2022-05-31T15:58:33,002][WARN ][o.e.d.PeerFinder         ] [es-node05] address [10.120.0.22:9300], node [null], requesting [false] connection failed: [][10.120.0.22:9300] connect_exception: No route to host: /10.120.0.22:9300: No route to host
[2022-05-31T15:58:33,003][WARN ][o.e.d.PeerFinder         ] [es-node05] address [10.120.0.24:9300], node [null], requesting [false] connection failed: [][10.120.0.24:9300] connect_exception: No route to host: /10.120.0.24:9300: No route to host
[2022-05-31T15:58:33,003][WARN ][o.e.d.PeerFinder         ] [es-node05] address [10.120.0.31:9300], node [null], requesting [false] connection failed: [][10.120.0.31:9300] connect_exception: No route to host: /10.120.0.31:9300: No route to host
[2022-05-31T15:58:33,020][WARN ][o.e.d.PeerFinder         ] [es-node05] address [10.120.0.8:9300], node [null], requesting [false] connection failed: [es-node04][10.120.0.8:9300] non-master-eligible node found
[2022-05-31T15:58:34,020][WARN ][o.e.d.PeerFinder         ] [es-node05] address [10.120.0.8:9300], node [null], requesting [false] connection failed: [es-node04][10.120.0.8:9300] non-master-eligible node found
[2022-05-31T15:58:35,020][WARN ][o.e.d.PeerFinder         ] [es-node05] address [10.120.0.8:9300], node [null], requesting [false] connection failed: [es-node04][10.120.0.8:9300] non-master-eligible node found
[2022-05-31T15:58:36,020][WARN ][o.e.d.PeerFinder         ] [es-node05] address [10.120.0.8:9300], node [null], requesting [false] connection failed: [es-node04][10.120.0.8:9300] non-master-eligible node found
[2022-05-31T15:58:36,074][WARN ][o.e.d.PeerFinder         ] [es-node05] address [10.120.0.7:9300], node [null], requesting [false] connection failed: [][10.120.0.7:9300] connect_exception: No route to host: /10.120.0.7:9300: No route to host
[2022-05-31T15:58:36,074][WARN ][o.e.d.PeerFinder         ] [es-node05] address [10.120.0.22:9300], node [null], requesting [false] connection failed: [][10.120.0.22:9300] connect_exception: No route to host: /10.120.0.22:9300: No route to host

I have 5 nodes, 3 are master, 2 are data nodes, I just modified the port, the master has succeeded, but the log of the data node still has errors.

cluster.name: es
node.name: es-node04
node.attr.box_type: cold
node.attr.size: 4T
node.attr.location: ydy_cluster
node.roles: [ data_cold, data_content, data, remote_cluster_client ]

path.data: /home/elasticsearch/esdata/
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: ["10.120.0.7:9300","10.120.0.24:9300","10.120.0.31:9300","10.120.0.8:9300","10.120.0.22:9300"]
#cluster.initial_master_nodes: ["es-node01","es-node02","es-node03"]


xpack.security.enabled: false
xpack.monitoring.enabled: true
xpack.monitoring.collection.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-stack-ca.p12
xpack.security.transport.ssl.truststore.path: elastic-stack-ca.p12
5

problem solved.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.