Version: 8.5.0
The cluster has 3 nodes:
node-001 (master)
node-002 (data)
node-003 (data)
The master node VM crashed and upon restarting the master node, the data nodes are not getting discovered. The cluster was running fine before
getting below error:
[2023-08-01T11:54:56,389][WARN ][o.e.d.HandshakingTransportAddressConnector] [node-003] completed handshake with [{node-001}{BdJx0VGYSyqUUN8BEhd9Ew}{q-wCbboEQ6m8Ge5DQCAIBQ}{node-001}{172.17.0.1}{172.17.0.1:9300}{dimr}] at [172.31.67.22:9300] but followup connection to [172.17.0.1:9300] failed
org.elasticsearch.transport.ConnectTransportException: [node-001][172.17.0.1:9300] handshake failed. unexpected remote node {node-003}{uKPcP22bQoSZALc5xj34cA}{SXuW6ZkqRj6nbplBzMjQNg}{node-003}{172.17.0.1}{172.17.0.1:9300}{dir}{zone=search,aggregation, xpack.installed=true}
at org.elasticsearch.transport.TransportService.lambda$connectionValidator$4(TransportService.java:462) ~[elasticsearch-8.5.0.jar:?]
at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:121) ~[elasticsearch-8.5.0.jar:?]
at org.elasticsearch.transport.TransportService.lambda$handshake$6(TransportService.java:560) ~[elasticsearch-8.5.0.jar:?]
at org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:245) ~[elasticsearch-8.5.0.jar:?]
at org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:43) ~[elasticsearch-8.5.0.jar:?]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1362) ~[elasticsearch-8.5.0.jar:?]
at org.elasticsearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:369) ~[elasticsearch-8.5.0.jar:?]
at org.elasticsearch.transport.InboundHandler$2.doRun(InboundHandler.java:361) ~[elasticsearch-8.5.0.jar:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:892) ~[elasticsearch-8.5.0.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.5.0.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1589) ~[?:?]