Data node not able to discover master node

Elastic Stack Elasticsearch
1

I have 1 master eligible node and 1 data node

Master config:

# ---------------------------------- Cluster -----------------------------------
cluster.name: ELS_Cluster
# ------------------------------------ Node ------------------------------------
node.name: els-pr-01
node.master: true
node.data: false

node.ingest: false

# ----------------------------------- Paths ------------------------------------
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
# ----------------------------------- Memory -----------------------------------
bootstrap.memory_lock: true
# ---------------------------------- Network -----------------------------------
network.host: els-pr-01
http.port: 9200
http.compression: true
http.cors.enabled: true
http.cors.allow-origin: '*'
http.cors.max-age: 86400
http.cors.allow-methods: OPTIONS, HEAD, GET, POST, PUT, DELETE
http.cors.allow-headers: X-Requested-With, Content-Type, Content-Length
http.cors.allow-credentials: true
# --------------------------------- Discovery ----------------------------------

cluster.initial_master_nodes: ["els-pr-01"]

discovery.seed_hosts: ["els-pr-01:9300"]
logger.org.elasticsearch.discovery: TRACE
# ---------------------------------- Various -----------------------------------
action.auto_create_index: true

Data node config:

# ---------------------------------- Cluster -----------------------------------
cluster.name: ELS_Cluster
# ------------------------------------ Node ------------------------------------
node.name: data01
node.master: false
node.data: true

# ----------------------------------- Paths ------------------------------------
path.data: /mnt/data/elasticsearch
path.logs: /mnt/data/elasticsearch/log
# ----------------------------------- Memory -----------------------------------
bootstrap.memory_lock: true
# ---------------------------------- Network -----------------------------------
network.host: els-pr-data01
http.port: 9200
http.compression: true
http.cors.enabled: true
http.cors.allow-origin: '*'
http.cors.max-age: 86400
http.cors.allow-methods: OPTIONS, HEAD, GET, POST, PUT, DELETE
http.cors.allow-headers: X-Requested-With, Content-Type, Content-Length
http.cors.allow-credentials: true
# --------------------------------- Discovery ----------------------------------

discovery.seed_hosts: ["els-pr-01:9300"]
#iscovery.zen.minimum_master_nodes: 1
logger.org.elasticsearch.discovery: TRACE
# ---------------------------------- Various -----------------------------------
action.auto_create_index: true

Trace logs:

[2020-09-22T12:33:31,025][TRACE][o.e.d.PeerFinder ] [data01] probing master nodes from cluster state: nodes:
{data01}{Vh8n93idSZmHQoudwDm5HQ}{D35qMH9NRmiBfMKkyulmxA}{dc-els-pr-data01}{127.0.1.1:9300}{dir}, local

[2020-09-22T12:33:31,026][TRACE][o.e.d.SeedHostsResolver ] [data01] resolved host [10.64.20.204:9300] to [10.64.20.204:9300]
[2020-09-22T12:33:31,026][TRACE][o.e.d.PeerFinder ] [data01] probing resolved transport addresses [10.64.20.204:9300]
[2020-09-22T12:33:31,026][TRACE][o.e.d.PeerFinder ] [data01] Peer{transportAddress=10.64.20.204:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
[2020-09-22T12:33:31,026][TRACE][o.e.d.HandshakingTransportAddressConnector] [data01] [connectToRemoteMasterNode[10.64.20.204:9300]] opening probe connection
[2020-09-22T12:33:31,030][DEBUG][o.e.d.PeerFinder ] [data01] Peer{transportAddress=10.64.20.204:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
org.elasticsearch.transport.ConnectTransportException: [10.64.20.204:9300] connect_exception
at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:966) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:198) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.8.0.jar:7.8.0]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2152) ~[?:?]
at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.8.0.jar:7.8.0]
at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$addListener$0(Netty4TcpChannel.java:68) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:570) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:549) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:608) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.64.20.204:9300
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
at sun.nio.ch.Net.pollConnectNow(Net.java:589) ~[?:?]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:839) ~[?:?]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) ~[?:?]
... 7 more

Not sure what am I doing wrong.

2

The connection was refused: this node cannot connect to the other node on 10.64.20.204:9300.

3

Thanks for the quick response, any common reasons why this might be happening?

4

its listening on master node

java 484632 elasticsearch 251u IPv6 1215324 0t0 TCP 127.0.1.1:9300 (LISTEN)

5

Looks like it's only listening on a local interface only, not 10.64.20.204? You need to set network.host to something that resolves to the right address.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.