Master not Discovered yet

Elastic Stack Elasticsearch
1

Hello community, I am facing an issue, I have a cluster consisting 3 nodes in NZ(Master-Data), one node in PK(Data Node) everything is up and running over the public IPs.
Now the issue comes when I try to join my new IR node(Data Node) with the existing cluster.

curl -X GET "https://localhost:9200/_cat/nodes" -u elastic -k
-------------
gives all the PK and NZ nodes that are part of cluster
-------------

cluster state on NZ and PK nodes
curl -XGET [https://localhost:9200](https://localhost:9200/) -k -u elastic
Output
----------------------------
Enter host password for user 'elastic':
{
  "name" : "elasticsearch-cluster",
  "cluster_name" : "cluster-name",
  "cluster_uuid" : "UUID_no",
  "version" : {
    "number" : "7.14.2",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "hash_here",
    "build_date" : "2021-09-15T10:18:09.722761972Z",
    "build_snapshot" : false,
    "lucene_version" : "8.9.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
-----------------------------

cluster state on IR node
curl -XGET [https://localhost:9200](https://localhost:9200/) -k -u elastic
Output
----------------------------
{"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]","header":{"WWW-Authenticate":["Bearer realm=\"security\"","ApiKey","Basic realm=\"security\" charset=\"UTF-8\""]}}],"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]","header":{"WWW-Authenticate":["Bearer realm=\"security\"","ApiKey","Basic realm=\"security\" charset=\"UTF-8\""]}},"status":401}[root@elasticsearch_IR nodes]#
----------------------------

Below I have attached the logs also;

[2022-01-28T12:15:44,718][WARN ][o.e.c.c.ClusterFormationFailureHelper] [elasticsearch_IR] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [ip-address-of-not-joining-node, ip-address-cluster-nodes, ip-address-cluster-nodes, ip-address-cluster-nodes] to bootstrap a cluster: have discovered [{elasticsearch_IR}{9zBzf4v_T8KZirfUVhcf6A}{emVNrY7STr6WibBu_bNRJg}{ip-address-of-not-joining-node}{ip-address-of-not-joining-node:9300}{cdfhimrstw}]; discovery will continue using [43.255.162.232:9300, 43.255.162.250:9300, 43.255.160.176:9300] from hosts providers and [{elasticsearch_IR}{9zBzf4v_T8KZirfUVhcf6A}{emVNrY7STr6WibBu_bNRJg}{ip-address-of-not-joining-node}{ip-address-of-not-joining-node:9300}{cdfhimrstw}] from last-known cluster state; node term 0, last-accepted version 0 in term 0

2

Please share the full logs covering at least 10 minutes after starting the new node. The single message you've shared tells us that there's a problem with discovery, but the details will be in other log messages.

3

almost same lines are repeating. some other logs are @DavidTurner

Caused by: java.nio.file.NoSuchFileException: /var/lib/elasticsearch/nodes/0/_state
java.io.UncheckedIOException: java.nio.file.NoSuchFileException: /var/lib/elasticsearch/nodes/0/_state
[2022-01-28T10:51:32,200][WARN ][o.e.d.HandshakingTransportAddressConnector] [elasticsearch_IR] [connectToRemoteMasterNode[Node-1_NZ:9300]] completed handshake with [{elasticsearch-valentia}{HL4NBLyYTDORSs4SctYf8A}{g2JTrKZQQ0GivDmywjeeOA}{10.150.100.121}{10.150.100.121:9300}{cdfhimstw}{country=nz, xpack.installed=true, transform.node=true}] but followup connection failed
org.elasticsearch.transport.ConnectTransportException: [elasticsearch-valentia][10.150.100.121:9300] connect_timeout[30s]
	at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onTimeout(TcpTransport.java:994) ~[elasticsearch-7.14.2.jar:7.14.2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:673) ~[elasticsearch-7.14.2.jar:7.14.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
	at java.lang.Thread.run(Thread.java:831) [?:?]
[2022-01-28T10:51:32,201][WARN ][o.e.d.PeerFinder         ] [elasticsearch_IR] address [Node-1_NZ:9300], node [null], requesting [false] connection failed: [elasticsearch-valentia][10.150.100.121:9300] connect_timeout[30s]
[2022-01-28T10:51:34,916][WARN ][o.e.c.c.ClusterFormationFailureHelper] [elasticsearch_IR] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [ip-addr-for-node-not-joining, Node-1_NZ, Node-2_NZ, Node-3_NZ] to bootstrap a cluster: have discovered [{elasticsearch_IR}{h_MJL2UjTYuZFwilTYOMDg}{wCumjlYVSxi-t49H-jmSlg}{ip-addr-for-node-not-joining}{ip-addr-for-node-not-joining:9300}{cdfhimrstw}]; discovery will continue using [Node-1_NZ:9300, Node-2_NZ:9300, Node-3_NZ:9300] from hosts providers and [{elasticsearch_IR}{h_MJL2UjTYuZFwilTYOMDg}{wCumjlYVSxi-t49H-jmSlg}{ip-addr-for-node-not-joining}{ip-addr-for-node-not-joining:9300}{cdfhimrstw}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2022-01-28T10:51:44,916][WARN ][o.e.c.c.ClusterFormationFailureHelper] [elasticsearch_IR] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [ip-addr-for-node-not-joining, Node-1_NZ, Node-2_NZ, Node-3_NZ] to bootstrap a cluster: have discovered [{elasticsearch_IR}{h_MJL2UjTYuZFwilTYOMDg}{wCumjlYVSxi-t49H-jmSlg}{ip-addr-for-node-not-joining}{ip-addr-for-node-not-joining:9300}{cdfhimrstw}]; discovery will continue using [Node-1_NZ:9300, Node-2_NZ:9300, Node-3_NZ:9300] from hosts providers and [{elasticsearch_IR}{h_MJL2UjTYuZFwilTYOMDg}{wCumjlYVSxi-t49H-jmSlg}{ip-addr-for-node-not-joining}{ip-addr-for-node-not-joining:9300}{cdfhimrstw}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2022-01-28T10:51:54,917][WARN ][o.e.c.c.ClusterFormationFailureHelper] [elasticsearch_IR] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [ip-addr-for-node-not-joining, Node-1_NZ, Node-2_NZ, Node-3_NZ] to bootstrap a cluster: have discovered [{elasticsearch_IR}{h_MJL2UjTYuZFwilTYOMDg}{wCumjlYVSxi-t49H-jmSlg}{ip-addr-for-node-not-joining}{ip-addr-for-node-not-joining:9300}{cdfhimrstw}]; discovery will continue using [Node-1_NZ:9300, Node-2_NZ:9300, Node-3_NZ:9300] from hosts providers and [{elasticsearch_IR}{h_MJL2UjTYuZFwilTYOMDg}{wCumjlYVSxi-t49H-jmSlg}{ip-addr-for-node-not-joining}{ip-addr-for-node-not-joining:9300}{cdfhimrstw}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
4

It's timing out connecting to 10.150.100.121:9300. You need to ensure that this node can access that address.

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.