Getting the Warning message as master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes

Elastic Stack Elasticsearch
1

I'm trying to create Elasticsearch cluster with 9 nodes. i'm working on Elastic 7.3.2 version .i have 3 master only nodes , 1 voting only nodes, rest are date node,cordinate node and ingest node. i am getting the error as "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes"
The error i am getting is

2019-10-23T04:14:58,906][WARN ][o.e.c.c.ClusterFormationFailureHelper] [master-a] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [master-a, master-b, master-c] to bootstrap a cluster: have discovered [{master-a}{8RPo7rbGTyaImEW7riET6Q}{8PSk648MTJePxNMKHuCcUg}{10.55.50.104}{10.55.50.104:9300}{m}{xpack.installed=true}, {master-b}{8RPo7rbGTyaImEW7riET6Q}{EzAjKx6gQHyMeuwHkIqtAA}{10.55.50.101}{10.55.50.101:9300}{m}{xpack.installed=true}, {vote}{8RPo7rbGTyaImEW7riET6Q}{O442bjqIRZW4h1UC1MPpSw}{10.55.50.108}{10.55.50.108:9300}{mv}{xpack.installed=true}]; discovery will continue using [10.55.50.100:9300, 10.55.50.101:9300, 10.55.50.102:9300, 10.55.50.103:9300, 10.55.50.105:9300, 10.55.50.106:9300, 10.55.50.107:9300, 10.55.50.108:9300] from hosts providers and [{master-a}{8RPo7rbGTyaImEW7riET6Q}{8PSk648MTJePxNMKHuCcUg}{10.55.50.104}{10.55.50.104:9300}{m}{xpack.installed=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2019-10-23T04:15:02,177][WARN ][o.e.c.c.ClusterBootstrapService] [master-a] exception when bootstrapping with VotingConfiguration{{bootstrap-placeholder}-master-c,8RPo7rbGTyaImEW7riET6Q}, rescheduling.

2

Hi @Ozil, it says "exception when bootstrapping" but you have not shared the details of the exception (the following block of lines). What do they say?

3

Actually, I see the issue:

Your three nodes all have the same node ID. This means you copied their data paths. You should not do this. Each node should start from a completely empty data path.

1 Like
4

Thanks for your quick reply....for that do i need to clear all the data with in the data path ?

5

Yes that's right.

6

I just cleared the data with in the data path and i started again still cluster not forming.in yml file i clearly mentioned cluster.initial_master_nodes (maste-a,master-b and master-b)

now i am getting the logs like this

[2019-10-23T08:24:35,254][DEBUG][o.e.d.PeerFinder         ] [master-a] Peer{transportAddress=10.55.50.103:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
org.elasticsearch.transport.ConnectTransportException: [data2][10.55.50.103:9300] non-master-eligible node found
        at org.elasticsearch.discovery.HandshakingTransportAddressConnector$1.doRun(HandshakingTransportAddressConnector.java:109) [elasticsearch-7.3.2.jar:7.3.2]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.3.2.jar:7.3.2]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.2.jar:7.3.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:834) [?:?]
[2019-10-23T08:24:35,254][TRACE][o.e.d.HandshakingTransportAddressConnector] [master-a] [connectToRemoteMasterNode[10.55.50.105:9300]] opened probe connection
[2019-10-23T08:24:35,254][TRACE][o.e.d.HandshakingTransportAddressConnector] [master-a] [connectToRemoteMasterNode[10.55.50.100:9300]] opened probe connection
[2019-10-23T08:24:35,255][TRACE][o.e.d.HandshakingTransportAddressConnector] [master-a] [connectToRemoteMasterNode[10.55.50.105:9300]] handshake successful: {load_balancer2}{8RPo7rbGTyaImEW7riET6Q}{G22Mw6keR4GNHQM-WaQN-g}{10.55.50.105}{10.55.50.105:9300}{xpack.installed=true}
[2019-10-23T08:24:35,255][TRACE][o.e.d.HandshakingTransportAddressConnector] [master-a] [connectToRemoteMasterNode[10.55.50.100:9300]] handshake successful: {load_balancer}{8RPo7rbGTyaImEW7riET6Q}{l02dQxfQRSqPvm912LhBaQ}{10.55.50.100}{10.55.50.100:9300}{xpack.installed=true}
[2019-10-23T08:24:35,255][DEBUG][o.e.d.PeerFinder         ] [master-a] Peer{transportAddress=10.55.50.100:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
org.elasticsearch.transport.ConnectTransportException: [load_balancer][10.55.50.100:9300] non-master-eligible node found
        at org.elasticsearch.discovery.HandshakingTransportAddressConnector$1.doRun(HandshakingTransportAddressConnector.java:109) [elasticsearch-7.3.2.jar:7.3.2]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.3.2.jar:7.3.2]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.2.jar:7.3.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:834) [?:?]
[2019-10-23T08:24:35,255][DEBUG][o.e.d.PeerFinder         ] [master-a] Peer{transportAddress=10.55.50.105:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
org.elasticsearch.transport.ConnectTransportException: [load_balancer2][10.55.50.105:9300] non-master-eligible node found
        at org.elasticsearch.discovery.HandshakingTransportAddressConnector$1.doRun(HandshakingTransportAddressConnector.java:109) [elasticsearch-7.3.2.jar:7.3.2]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.3.2.jar:7.3.2]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.2.jar:7.3.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:834) [?:?]
[2019-10-23T08:24:35,256][TRACE][o.e.d.HandshakingTransportAddressConnector] [master-a] [connectToRemoteMasterNode[10.55.50.106:9300]] handshake successful: {cordinate}{8RPo7rbGTyaImEW7riET6Q}{1vQ8M_5XQKWjbLUyva3JDA}{10.55.50.106}{10.55.50.106:9300}{i}{xpack.installed=true}
[2019-10-23T08:24:35,256][DEBUG][o.e.d.PeerFinder         ] [master-a] Peer{transportAddress=10.55.50.106:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
org.elasticsearch.transport.ConnectTransportException: [cordinate][10.55.50.106:9300] non-master-eligible node found
        at org.elasticsearch.discovery.HandshakingTransportAddressConnector$1.doRun(HandshakingTransportAddressConnector.java:109) [elasticsearch-7.3.2.jar:7.3.2]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.3.2.jar:7.3.2]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.2.jar:7.3.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:834) [?:?]
[2019-10-23T08:24:35,272][TRACE][o.e.d.PeerFinder         ] [master-a] startProbe(10.55.50.104:9300) not probing local node
[2019-10-23T08:24:35,283][TRACE][o.e.d.PeerFinder         ] [master-a] startProbe(10.55.50.104:9300) not probing local node
[2019-10-23T08:24:35,303][TRACE][o.e.d.PeerFinder         ] [master-a] startProbe(10.55.50.104:9300) not probing local node
[2019-10-23T08:24:35,360][TRACE][o.e.d.PeerFinder         ] [master-a] startProbe(10.55.50.104:9300) not probing local node
[2019-10-23T08:24:35,664][TRACE][o.e.d.PeerFinder         ] [master-a] startProbe(10.55.50.104:9300) not probing local node
[2019-10-23T08:24:35,833][TRACE][o.e.d.PeerFinder         ] [master-a] startProbe(10.55.50.104:9300) not probing local node
[2019-10-23T08:24:35,877][TRACE][o.e.d.PeerFinder         ] [master-a] startProbe(10.55.50.104:9300) not probing local node
[2019-10-23T08:24:36,070][TRACE][o.e.d.PeerFinder         ] [master-a] startProbe(10.55.50.104:9300) not probing local node
[2019-10-23T08:24:36,249][TRACE][o.e.d.PeerFinder         ] [master-a] Peer{transportAddress=10.55.50.101:9300, discoveryNode={master-b}{8RPo7rbGTyaImEW7riET6Q}{QyhzAWu9TnemuYgG3XUOjA}{10.55.50.101}{10.55.50.101:9300}{m}{xpack.installed=true}, peersRequestInFlight=false} requesting peers
[2019-10-23T08:24:36,249][TRACE][o.e.d.PeerFinder         ] [master-a] Peer{transportAddress=10.55.50.107:9300, discoveryNode={master-c}{8RPo7rbGTyaImEW7riET6Q}{RCMv2M0LQnKeXHq-WszR7w}{10.55.50.107}{10.55.50.107:9300}{m}{xpack.installed=true}, peersRequestInFlight=false} requesting peers
7

After clearing the data from the data path ..still getting the the nodes with same ID.

8

I can only see DEBUG and TRACE messages in the log you have shared, so nothing to worry about there.

The node ID is stored in the data path, and generated randomly if the data path is empty. If you are still getting duplicate node IDs then I suspect you are not using empty data paths.

1 Like
9

Thank you @DavidTurner. it worked. now my cluster is up and running as expected.

1 Like
10

The way to bootstrap cluster has changed recently. Please have a look here https://www.elastic.co/guide/en/elasticsearch/reference/7.3/modules-discovery-bootstrap-cluster.html

11

That's not really relevant here @slavo, duplicate node IDs would prevent you forming a cluster in older versions too.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.