Data node not able to discover master node

I have 1 master eligible node and 1 data node

Master config:

# ---------------------------------- Cluster -----------------------------------
cluster.name: ELS_Cluster
# ------------------------------------ Node ------------------------------------
node.name: els-pr-01
node.master: true
node.data: false

node.ingest: false

# ----------------------------------- Paths ------------------------------------
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
# ----------------------------------- Memory -----------------------------------
bootstrap.memory_lock: true
# ---------------------------------- Network -----------------------------------
network.host: els-pr-01
http.port: 9200
http.compression: true
http.cors.enabled: true
http.cors.allow-origin: '*'
http.cors.max-age: 86400
http.cors.allow-methods: OPTIONS, HEAD, GET, POST, PUT, DELETE
http.cors.allow-headers: X-Requested-With, Content-Type, Content-Length
http.cors.allow-credentials: true
# --------------------------------- Discovery ----------------------------------

cluster.initial_master_nodes: ["els-pr-01"]

discovery.seed_hosts: ["els-pr-01:9300"]
logger.org.elasticsearch.discovery: TRACE
# ---------------------------------- Various -----------------------------------
action.auto_create_index: true

Data node config:

# ---------------------------------- Cluster -----------------------------------
cluster.name: ELS_Cluster
# ------------------------------------ Node ------------------------------------
node.name: data01
node.master: false
node.data: true

# ----------------------------------- Paths ------------------------------------
path.data: /mnt/data/elasticsearch
path.logs: /mnt/data/elasticsearch/log
# ----------------------------------- Memory -----------------------------------
bootstrap.memory_lock: true
# ---------------------------------- Network -----------------------------------
network.host: els-pr-data01
http.port: 9200
http.compression: true
http.cors.enabled: true
http.cors.allow-origin: '*'
http.cors.max-age: 86400
http.cors.allow-methods: OPTIONS, HEAD, GET, POST, PUT, DELETE
http.cors.allow-headers: X-Requested-With, Content-Type, Content-Length
http.cors.allow-credentials: true
# --------------------------------- Discovery ----------------------------------

discovery.seed_hosts: ["els-pr-01:9300"]
#iscovery.zen.minimum_master_nodes: 1
logger.org.elasticsearch.discovery: TRACE
# ---------------------------------- Various -----------------------------------
action.auto_create_index: true

Trace logs:

[2020-09-22T12:33:31,025][TRACE][o.e.d.PeerFinder ] [data01] probing master nodes from cluster state: nodes:
{data01}{Vh8n93idSZmHQoudwDm5HQ}{D35qMH9NRmiBfMKkyulmxA}{dc-els-pr-data01}{127.0.1.1:9300}{dir}, local

[2020-09-22T12:33:31,026][TRACE][o.e.d.SeedHostsResolver ] [data01] resolved host [10.64.20.204:9300] to [10.64.20.204:9300]
[2020-09-22T12:33:31,026][TRACE][o.e.d.PeerFinder ] [data01] probing resolved transport addresses [10.64.20.204:9300]
[2020-09-22T12:33:31,026][TRACE][o.e.d.PeerFinder ] [data01] Peer{transportAddress=10.64.20.204:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
[2020-09-22T12:33:31,026][TRACE][o.e.d.HandshakingTransportAddressConnector] [data01] [connectToRemoteMasterNode[10.64.20.204:9300]] opening probe connection
[2020-09-22T12:33:31,030][DEBUG][o.e.d.PeerFinder ] [data01] Peer{transportAddress=10.64.20.204:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
org.elasticsearch.transport.ConnectTransportException: [10.64.20.204:9300] connect_exception
at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:966) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:198) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.8.0.jar:7.8.0]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2152) ~[?:?]
at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.8.0.jar:7.8.0]
at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$addListener$0(Netty4TcpChannel.java:68) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:570) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:549) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:608) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.64.20.204:9300
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
at sun.nio.ch.Net.pollConnectNow(Net.java:589) ~[?:?]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:839) ~[?:?]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) ~[?:?]
... 7 more

Not sure what am I doing wrong.

The connection was refused: this node cannot connect to the other node on 10.64.20.204:9300.

Thanks for the quick response, any common reasons why this might be happening?

its listening on master node

java 484632 elasticsearch 251u IPv6 1215324 0t0 TCP 127.0.1.1:9300 (LISTEN)

Looks like it's only listening on a local interface only, not 10.64.20.204? You need to set network.host to something that resolves to the right address.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.