Master not discovered or elected yet, an election requires at least 2 nodes

Elastic Stack Elasticsearch
4

I built a cluster of 3 nodes with elasticsearch-7.1 two weeks ago.
my commnds :

  • ./elasticsearch -E node.name=node1 -E cluster.name=my-app -E path.data=/datas/es7/node1_data -E path.logs=/logs/node1_log -d

  • ./elasticsearch -E node.name=node2 -E cluster.name=my-app -E path.data=/datas/es7/node2_data -E path.logs=/logs/node2_log -d

  • ./elasticsearch -E node.name=node3 -E cluster.name=my-app -E path.data=/es7/node3_data -E path.logs=/logs/node3_log -d

now,I shutdown all nodes, and I want to start the cluster with only one node : node1. Unfortunately , it failed.here is the log:

[2019-12-31T14:29:23,954][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node1] 
master not discovered or elected yet, an election requires at least 2 nodes with 
ids from [5e3LlD8QTHmGaJDhlKY-8w, cJyHBa9ATf-2vg8Tnt7KeQ, JHVEwFfkStOI6qG6QawqkA], 
have discovered [] which is not a quorum; discovery will continue using 
[127.0.0.1:9300, 127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304] 
from hosts providers and [{node1}{JHVEwFfkStOI6qG6QawqkA}{tv4ieuTOQn6wW9fUGwNXzg}{172.21.0.1}{172.21.0.1:9300}
{ml.machine_memory=3975200768, xpack.installed=true, ml.max_open_jobs=20}] 
from last-known cluster state; node term 8, last-accepted version 533 in term 8

and my elasticsearch.yaml

  • cluster.name: my-app

  • network.host: 0.0.0.0

  • node.name: node1

  • path.data: /datas/es7/node1_data

  • path.logs: /logs/es7/node1_log

and I try to set

cluster.initial_master_nodes: ["node1"]

but it makes no difference .
here is logs when I set logger.org.elasticsearch.discovery: TRACE

[2019-12-31T14:55:04,331][INFO ][o.e.n.Node               ] [node1] starting ...

[2019-12-31T14:55:04,468][INFO ][o.e.t.TransportService   ] [node1] publish_address {172.21.0.1:9300}, bound_addresses {0.0.0.0:9300}
[2019-12-31T14:55:04,479][INFO ][o.e.b.BootstrapChecks    ] [node1] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-12-31T14:55:04,520][DEBUG][o.e.d.SeedHostsResolver  ] [node1] using max_concurrent_resolvers [10], resolver timeout [5s]
[2019-12-31T14:55:04,520][INFO ][o.e.c.c.Coordinator      ] [node1] cluster UUID [G4ku2BYQRIG-dHdIDQ7tvA]
[2019-12-31T14:55:04,524][TRACE][o.e.d.PeerFinder         ] [node1] activating with nodes: 
   {node1}{JHVEwFfkStOI6qG6QawqkA}{-FVunPWUSIGpB5ndu00tfw}{172.21.0.1}{172.21.0.1:9300}{ml.machine_memory=3975200768, xpack.installed=true, ml.max_open_jobs=20}, local

[2019-12-31T14:55:04,525][TRACE][o.e.d.PeerFinder         ] [node1] probing master nodes from cluster state: nodes: 
   {node1}{JHVEwFfkStOI6qG6QawqkA}{-FVunPWUSIGpB5ndu00tfw}{172.21.0.1}{172.21.0.1:9300}{ml.machine_memory=3975200768, xpack.installed=true, ml.max_open_jobs=20}, local

[2019-12-31T14:55:04,525][TRACE][o.e.d.PeerFinder         ] [node1] startProbe(172.21.0.1:9300) not probing local node
[2019-12-31T14:55:04,537][TRACE][o.e.d.SeedHostsResolver  ] [node1] resolved host [127.0.0.1] to [127.0.0.1:9300, 127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304]
[2019-12-31T14:55:04,538][TRACE][o.e.d.PeerFinder         ] [node1] probing resolved transport addresses [127.0.0.1:9300, 127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304]
[2019-12-31T14:55:04,539][TRACE][o.e.d.PeerFinder         ] [node1] Peer{transportAddress=127.0.0.1:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
[2019-12-31T14:55:04,546][TRACE][o.e.d.PeerFinder         ] [node1] Peer{transportAddress=127.0.0.1:9301, discoveryNode=null, peersRequestInFlight=false} attempting connection
[2019-12-31T14:55:04,546][TRACE][o.e.d.PeerFinder         ] [node1] Peer{transportAddress=127.0.0.1:9302, discoveryNode=null, peersRequestInFlight=false} attempting connection
[2019-12-31T14:55:04,546][TRACE][o.e.d.PeerFinder         ] [node1] Peer{transportAddress=127.0.0.1:9303, discoveryNode=null, peersRequestInFlight=false} attempting connection
[2019-12-31T14:55:04,549][TRACE][o.e.d.PeerFinder         ] [node1] Peer{transportAddress=127.0.0.1:9304, discoveryNode=null, peersRequestInFlight=false} attempting connection
[2019-12-31T14:55:04,551][TRACE][o.e.d.HandshakingTransportAddressConnector] [node1] [connectToRemoteMasterNode[127.0.0.1:9303]] opening probe connection
[2019-12-31T14:55:04,553][TRACE][o.e.d.HandshakingTransportAddressConnector] [node1] [connectToRemoteMasterNode[127.0.0.1:9300]] opening probe connection
[2019-12-31T14:55:04,553][TRACE][o.e.d.HandshakingTransportAddressConnector] [node1] [connectToRemoteMasterNode[127.0.0.1:9302]] opening probe connection
[2019-12-31T14:55:04,554][TRACE][o.e.d.HandshakingTransportAddressConnector] [node1] [connectToRemoteMasterNode[127.0.0.1:9301]] opening probe connection
[2019-12-31T14:55:04,559][TRACE][o.e.d.HandshakingTransportAddressConnector] [node1] [connectToRemoteMasterNode[127.0.0.1:9304]] opening probe connection

[2019-12-31T14:55:04,600][DEBUG][o.e.d.PeerFinder         ] [node1] Peer{transportAddress=127.0.0.1:9304, discoveryNode=null, peersRequestInFlight=false} connection failed
org.elasticsearch.transport.ConnectTransportException: [][127.0.0.1:9304] connect_exception
	at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
	at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
	at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_161]
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_161]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_161]
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_161]
	at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
	at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:9304
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
	at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
	... 6 more
Caused by: java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
	at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
	... 6 more
[2019-12-31T14:55:04,655][TRACE][o.e.d.HandshakingTransportAddressConnector] [node1] [connectToRemoteMasterNode[127.0.0.1:9300]] opened probe connection
[2019-12-31T14:55:04,621][DEBUG][o.e.d.PeerFinder         ] [node1] Peer{transportAddress=127.0.0.1:9301, discoveryNode=null, peersRequestInFlight=false} connection failed
org.elasticsearch.transport.ConnectTransportException: [][127.0.0.1:9301] connect_exception.

so ,waht can I do to start the cluster with only one node ? and why "an election requires at least 2 nodes" when i alreay set cluster.initial_master_nodes: ["node1"]
Thanks

1 Like