Master not discovered or elected yet, an election requires at least 2 nodes

Elastic Stack Elasticsearch
1

[2019-12-31T14:29:23,954][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node1]
master not discovered or elected yet, an election requires at least 2 nodes with
ids from [5e3LlD8QTHmGaJDhlKY-8w, cJyHBa9ATf-2vg8Tnt7KeQ, JHVEwFfkStOI6qG6QawqkA],
have discovered which is not a quorum; discovery will continue using
[127.0.0.1:9300, 127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304]
from hosts providers and [{node1}{JHVEwFfkStOI6qG6QawqkA}{tv4ieuTOQn6wW9fUGwNXzg}{172.21.0.1}{172.21.0.1:9300}
{ml.machine_memory=3975200768, xpack.installed=true, ml.max_open_jobs=20}]
from last-known cluster state; node term 8, last-accepted version 533 in term 8

I built a cluster of 3 nodes with elasticsearch-7.1 two weeks ago.

./elasticsearch -E node.name=node1 -E cluster.name=my-app -E path.data=/datas/es7/node1_data -E path.logs=/logs/node1_log -d
./elasticsearch -E node.name=node2 -E cluster.name=my-app -E path.data=/datas/es7/node2_data -E path.logs=/logs/node2_log -d
./elasticsearch -E node.name=node3 -E cluster.name=my-app -E path.data=/es7/node3_data -E path.logs=/logs/node3_log -d

now,I shutdown all nodes, and I want to start the cluster with only one node : node1.
Unfortunately , it failed.
here is the log:

[2019-12-31T14:29:23,954][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node1]
master not discovered or elected yet, an election requires at least 2 nodes with
ids from [5e3LlD8QTHmGaJDhlKY-8w, cJyHBa9ATf-2vg8Tnt7KeQ, JHVEwFfkStOI6qG6QawqkA],
have discovered which is not a quorum; discovery will continue using
[127.0.0.1:9300, 127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304]
from hosts providers and [{node1}{JHVEwFfkStOI6qG6QawqkA}{tv4ieuTOQn6wW9fUGwNXzg}{172.21.0.1}{172.21.0.1:9300}
{ml.machine_memory=3975200768, xpack.installed=true, ml.max_open_jobs=20}]
from last-known cluster state; node term 8, last-accepted version 533 in term 8

here is the elasticsearch.ymal:

cluster.name: my-app
network.host: 0.0.0.0
node.name: node1
path.data: /datas/es7/node1_data
path.logs: /logs/es7/node1_log

and I try to set cluster.initial_master_nodes: ["node1"],it makes no difference .

here is logs when I set logger.org.elasticsearch.discovery: TRACE

[2019-12-31T14:55:04,331][INFO ][o.e.n.Node ] [node1] starting ...

[2019-12-31T14:55:04,468][INFO ][o.e.t.TransportService ] [node1] publish_address {172.21.0.1:9300}, bound_addresses {0.0.0.0:9300}
[2019-12-31T14:55:04,479][INFO ][o.e.b.BootstrapChecks ] [node1] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-12-31T14:55:04,520][DEBUG][o.e.d.SeedHostsResolver ] [node1] using max_concurrent_resolvers [10], resolver timeout [5s]
[2019-12-31T14:55:04,520][INFO ][o.e.c.c.Coordinator ] [node1] cluster UUID [G4ku2BYQRIG-dHdIDQ7tvA]
[2019-12-31T14:55:04,524][TRACE][o.e.d.PeerFinder ] [node1] activating with nodes:
{node1}{JHVEwFfkStOI6qG6QawqkA}{-FVunPWUSIGpB5ndu00tfw}{172.21.0.1}{172.21.0.1:9300}{ml.machine_memory=3975200768, xpack.installed=true, ml.max_open_jobs=20}, local

[2019-12-31T14:55:04,525][TRACE][o.e.d.PeerFinder ] [node1] probing master nodes from cluster state: nodes:
{node1}{JHVEwFfkStOI6qG6QawqkA}{-FVunPWUSIGpB5ndu00tfw}{172.21.0.1}{172.21.0.1:9300}{ml.machine_memory=3975200768, xpack.installed=true, ml.max_open_jobs=20}, local

[2019-12-31T14:55:04,525][TRACE][o.e.d.PeerFinder ] [node1] startProbe(172.21.0.1:9300) not probing local node
[2019-12-31T14:55:04,537][TRACE][o.e.d.SeedHostsResolver ] [node1] resolved host [127.0.0.1] to [127.0.0.1:9300, 127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304]
[2019-12-31T14:55:04,538][TRACE][o.e.d.PeerFinder ] [node1] probing resolved transport addresses [127.0.0.1:9300, 127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304]
[2019-12-31T14:55:04,539][TRACE][o.e.d.PeerFinder ] [node1] Peer{transportAddress=127.0.0.1:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
[2019-12-31T14:55:04,546][TRACE][o.e.d.PeerFinder ] [node1] Peer{transportAddress=127.0.0.1:9301, discoveryNode=null, peersRequestInFlight=false} attempting connection
[2019-12-31T14:55:04,546][TRACE][o.e.d.PeerFinder ] [node1] Peer{transportAddress=127.0.0.1:9302, discoveryNode=null, peersRequestInFlight=false} attempting connection
[2019-12-31T14:55:04,546][TRACE][o.e.d.PeerFinder ] [node1] Peer{transportAddress=127.0.0.1:9303, discoveryNode=null, peersRequestInFlight=false} attempting connection
[2019-12-31T14:55:04,549][TRACE][o.e.d.PeerFinder ] [node1] Peer{transportAddress=127.0.0.1:9304, discoveryNode=null, peersRequestInFlight=false} attempting connection
[2019-12-31T14:55:04,551][TRACE][o.e.d.HandshakingTransportAddressConnector] [node1] [connectToRemoteMasterNode[127.0.0.1:9303]] opening probe connection
[2019-12-31T14:55:04,553][TRACE][o.e.d.HandshakingTransportAddressConnector] [node1] [connectToRemoteMasterNode[127.0.0.1:9300]] opening probe connection
[2019-12-31T14:55:04,553][TRACE][o.e.d.HandshakingTransportAddressConnector] [node1] [connectToRemoteMasterNode[127.0.0.1:9302]] opening probe connection
[2019-12-31T14:55:04,554][TRACE][o.e.d.HandshakingTransportAddressConnector] [node1] [connectToRemoteMasterNode[127.0.0.1:9301]] opening probe connection
[2019-12-31T14:55:04,559][TRACE][o.e.d.HandshakingTransportAddressConnector] [node1] [connectToRemoteMasterNode[127.0.0.1:9304]] opening probe connection

[2019-12-31T14:55:04,600][DEBUG][o.e.d.PeerFinder ] [node1] Peer{transportAddress=127.0.0.1:9304, discoveryNode=null, peersRequestInFlight=false} connection failed
org.elasticsearch.transport.ConnectTransportException: [127.0.0.1:9304] connect_exception
at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_161]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_161]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_161]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_161]
at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:9304
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
... 6 more
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
... 6 more
[2019-12-31T14:55:04,655][TRACE][o.e.d.HandshakingTransportAddressConnector] [node1] [connectToRemoteMasterNode[127.0.0.1:9300]] opened probe connection
[2019-12-31T14:55:04,621][DEBUG][o.e.d.PeerFinder ] [node1] Peer{transportAddress=127.0.0.1:9301, discoveryNode=null, peersRequestInFlight=false} connection failed
org.elasticsearch.transport.ConnectTransportException: [127.0.0.1:9301] connect_exception
......

so ,waht can I do to start the cluster with only one node ?
and why "an election requires at least 2 nodes" when i alreay set cluster.initial_master_nodes: ["node1"]
Thanks

2

Your post is almost completely unreadable due to the lack of formatting. It's more likely to receive a useful response if you format it properly, particularly using the </> button for logs.

4

I built a cluster of 3 nodes with elasticsearch-7.1 two weeks ago.
my commnds :

  • ./elasticsearch -E node.name=node1 -E cluster.name=my-app -E path.data=/datas/es7/node1_data -E path.logs=/logs/node1_log -d

  • ./elasticsearch -E node.name=node2 -E cluster.name=my-app -E path.data=/datas/es7/node2_data -E path.logs=/logs/node2_log -d

  • ./elasticsearch -E node.name=node3 -E cluster.name=my-app -E path.data=/es7/node3_data -E path.logs=/logs/node3_log -d

now,I shutdown all nodes, and I want to start the cluster with only one node : node1. Unfortunately , it failed.here is the log:

[2019-12-31T14:29:23,954][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node1] 
master not discovered or elected yet, an election requires at least 2 nodes with 
ids from [5e3LlD8QTHmGaJDhlKY-8w, cJyHBa9ATf-2vg8Tnt7KeQ, JHVEwFfkStOI6qG6QawqkA], 
have discovered [] which is not a quorum; discovery will continue using 
[127.0.0.1:9300, 127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304] 
from hosts providers and [{node1}{JHVEwFfkStOI6qG6QawqkA}{tv4ieuTOQn6wW9fUGwNXzg}{172.21.0.1}{172.21.0.1:9300}
{ml.machine_memory=3975200768, xpack.installed=true, ml.max_open_jobs=20}] 
from last-known cluster state; node term 8, last-accepted version 533 in term 8

and my elasticsearch.yaml

  • cluster.name: my-app

  • network.host: 0.0.0.0

  • node.name: node1

  • path.data: /datas/es7/node1_data

  • path.logs: /logs/es7/node1_log

and I try to set

cluster.initial_master_nodes: ["node1"]

but it makes no difference .
here is logs when I set logger.org.elasticsearch.discovery: TRACE

[2019-12-31T14:55:04,331][INFO ][o.e.n.Node               ] [node1] starting ...

[2019-12-31T14:55:04,468][INFO ][o.e.t.TransportService   ] [node1] publish_address {172.21.0.1:9300}, bound_addresses {0.0.0.0:9300}
[2019-12-31T14:55:04,479][INFO ][o.e.b.BootstrapChecks    ] [node1] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-12-31T14:55:04,520][DEBUG][o.e.d.SeedHostsResolver  ] [node1] using max_concurrent_resolvers [10], resolver timeout [5s]
[2019-12-31T14:55:04,520][INFO ][o.e.c.c.Coordinator      ] [node1] cluster UUID [G4ku2BYQRIG-dHdIDQ7tvA]
[2019-12-31T14:55:04,524][TRACE][o.e.d.PeerFinder         ] [node1] activating with nodes: 
   {node1}{JHVEwFfkStOI6qG6QawqkA}{-FVunPWUSIGpB5ndu00tfw}{172.21.0.1}{172.21.0.1:9300}{ml.machine_memory=3975200768, xpack.installed=true, ml.max_open_jobs=20}, local

[2019-12-31T14:55:04,525][TRACE][o.e.d.PeerFinder         ] [node1] probing master nodes from cluster state: nodes: 
   {node1}{JHVEwFfkStOI6qG6QawqkA}{-FVunPWUSIGpB5ndu00tfw}{172.21.0.1}{172.21.0.1:9300}{ml.machine_memory=3975200768, xpack.installed=true, ml.max_open_jobs=20}, local

[2019-12-31T14:55:04,525][TRACE][o.e.d.PeerFinder         ] [node1] startProbe(172.21.0.1:9300) not probing local node
[2019-12-31T14:55:04,537][TRACE][o.e.d.SeedHostsResolver  ] [node1] resolved host [127.0.0.1] to [127.0.0.1:9300, 127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304]
[2019-12-31T14:55:04,538][TRACE][o.e.d.PeerFinder         ] [node1] probing resolved transport addresses [127.0.0.1:9300, 127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304]
[2019-12-31T14:55:04,539][TRACE][o.e.d.PeerFinder         ] [node1] Peer{transportAddress=127.0.0.1:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
[2019-12-31T14:55:04,546][TRACE][o.e.d.PeerFinder         ] [node1] Peer{transportAddress=127.0.0.1:9301, discoveryNode=null, peersRequestInFlight=false} attempting connection
[2019-12-31T14:55:04,546][TRACE][o.e.d.PeerFinder         ] [node1] Peer{transportAddress=127.0.0.1:9302, discoveryNode=null, peersRequestInFlight=false} attempting connection
[2019-12-31T14:55:04,546][TRACE][o.e.d.PeerFinder         ] [node1] Peer{transportAddress=127.0.0.1:9303, discoveryNode=null, peersRequestInFlight=false} attempting connection
[2019-12-31T14:55:04,549][TRACE][o.e.d.PeerFinder         ] [node1] Peer{transportAddress=127.0.0.1:9304, discoveryNode=null, peersRequestInFlight=false} attempting connection
[2019-12-31T14:55:04,551][TRACE][o.e.d.HandshakingTransportAddressConnector] [node1] [connectToRemoteMasterNode[127.0.0.1:9303]] opening probe connection
[2019-12-31T14:55:04,553][TRACE][o.e.d.HandshakingTransportAddressConnector] [node1] [connectToRemoteMasterNode[127.0.0.1:9300]] opening probe connection
[2019-12-31T14:55:04,553][TRACE][o.e.d.HandshakingTransportAddressConnector] [node1] [connectToRemoteMasterNode[127.0.0.1:9302]] opening probe connection
[2019-12-31T14:55:04,554][TRACE][o.e.d.HandshakingTransportAddressConnector] [node1] [connectToRemoteMasterNode[127.0.0.1:9301]] opening probe connection
[2019-12-31T14:55:04,559][TRACE][o.e.d.HandshakingTransportAddressConnector] [node1] [connectToRemoteMasterNode[127.0.0.1:9304]] opening probe connection

[2019-12-31T14:55:04,600][DEBUG][o.e.d.PeerFinder         ] [node1] Peer{transportAddress=127.0.0.1:9304, discoveryNode=null, peersRequestInFlight=false} connection failed
org.elasticsearch.transport.ConnectTransportException: [][127.0.0.1:9304] connect_exception
	at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
	at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
	at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_161]
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_161]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_161]
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_161]
	at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
	at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:9304
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
	at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
	... 6 more
Caused by: java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
	at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
	... 6 more
[2019-12-31T14:55:04,655][TRACE][o.e.d.HandshakingTransportAddressConnector] [node1] [connectToRemoteMasterNode[127.0.0.1:9300]] opened probe connection
[2019-12-31T14:55:04,621][DEBUG][o.e.d.PeerFinder         ] [node1] Peer{transportAddress=127.0.0.1:9301, discoveryNode=null, peersRequestInFlight=false} connection failed
org.elasticsearch.transport.ConnectTransportException: [][127.0.0.1:9301] connect_exception.

so ,waht can I do to start the cluster with only one node ? and why "an election requires at least 2 nodes" when i alreay set cluster.initial_master_nodes: ["node1"]
Thanks

1 Like
5

Thanks, that's much easier to read.

You are trying to remove two of your three master-eligible nodes (i.e. more than half of them), which requires extra steps to ensure that you do not lose data in the process.

This setting is ignored once the cluster has formed for the first time.

2 Likes
7

I see. You've been very helpful.Wish you all the best.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.