Master Node Logs:
[2022-10-20T09:04:58,943][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-1] removed {{elasticsearch-3}{AWs8jYiPR7u4jq_7duCQ9Q}{7wNFTeFyT-2926ZLRizZBw}{*.*.*.*}{*.*.*.*:9305}{ml.machine_memory=33505222656, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true},}, reason: apply cluster state (from master [master {elasticsearch-2}{eRxN_IvaSfSQ82zeaKw0LQ}{YELeTWyVS3u1h-hWXB0FrA}{*.*.*.*}{*.*.*.*:9305}{ml.machine_memory=33505226752, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} committed version [9]])
[2022-10-20T09:04:59,117][DEBUG][o.e.a.s.TransportSearchAction] [elasticsearch-1] All shards failed for phase: [query]
[2022-10-20T09:04:59,134][DEBUG][o.e.a.s.TransportSearchAction] [elasticsearch-1] All shards failed for phase: [query]
[2022-10-20T09:04:59,185][DEBUG][o.e.a.s.TransportSearchAction] [elasticsearch-1] All shards failed for phase: [query]
[2022-10-20T09:04:59,199][DEBUG][o.e.a.s.TransportSearchAction] [elasticsearch-1] All shards failed for phase: [query]
[2022-10-20T09:04:59,329][DEBUG][o.e.a.s.TransportSearchAction] [elasticsearch-1] All shards failed for phase: [query]
[2022-10-20T09:04:59,358][DEBUG][o.e.a.s.TransportSearchAction] [elasticsearch-1] All shards failed for phase: [query]
[2022-10-20T09:04:59,393][INFO ][o.e.d.z.ZenDiscovery ] [elasticsearch-1] master_left [{elasticsearch-2}{eRxN_IvaSfSQ82zeaKw0LQ}{YELeTWyVS3u1h-hWXB0FrA}{*.*.*.*}{*.*.*.*:9305}{ml.machine_memory=33505226752, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}], reason [shut_down]
[2022-10-20T09:04:59,393][DEBUG][o.e.a.s.TransportSearchAction] [elasticsearch-1] All shards failed for phase: [query]
[2022-10-20T09:04:59,393][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch-1] master left (reason = shut_down), current nodes: nodes:
{elasticsearch-1}{sZD70v3nSgK9y6MsfFvB2A}{6huHryUASNS8_z561jCgEA}{*.*.*.*}{*.*.*.*:9305}{ml.machine_memory=33505226752, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, local
{elasticsearch-2}{eRxN_IvaSfSQ82zeaKw0LQ}{YELeTWyVS3u1h-hWXB0FrA}{*.*.*.*}{*.*.*.*:9305}{ml.machine_memory=33505226752, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}, master
[2022-10-20T09:04:59,423][DEBUG][o.e.a.s.TransportSearchAction] [elasticsearch-1] All shards failed for phase: [query]
[2022-10-20T09:04:59,472][DEBUG][o.e.a.s.TransportSearchAction] [elasticsearch-1] All shards failed for phase: [query]
[2022-10-20T09:04:59,578][DEBUG][o.e.a.s.TransportSearchAction] [elasticsearch-1] All shards failed for phase: [query]
[2022-10-20T09:04:59,595][DEBUG][o.e.a.s.TransportSearchAction] [elasticsearch-1] All shards failed for phase: [query]
[2022-10-20T09:04:59,649][DEBUG][o.e.a.s.TransportSearchAction] [elasticsearch-1] All shards failed for phase: [query]
[2022-10-20T09:05:00,354][DEBUG][o.e.a.s.TransportSearchAction] [elasticsearch-1] All shards failed for phase: [query]
[2022-10-20T09:05:00,371][DEBUG][o.e.a.s.TransportSearchAction] [elasticsearch-1] All shards failed for phase: [query]
[2022-10-20T09:05:00,383][DEBUG][o.e.a.s.TransportSearchAction] [elasticsearch-1] All shards failed for phase: [query]
[2022-10-20T09:05:00,407][DEBUG][o.e.a.s.TransportSearchAction] [elasticsearch-1] All shards failed for phase: [query]
[2022-10-20T09:05:00,422][DEBUG][o.e.a.s.TransportSearchAction] [elasticsearch-1] All shards failed for phase: [query]
[2022-10-20T09:05:00,446][DEBUG][o.e.a.s.TransportSearchAction] [elasticsearch-1] All shards failed for phase: [query]
[2022-10-20T09:05:00,768][DEBUG][o.e.a.s.TransportSearchAction] [elasticsearch-1] All shards failed for phase: [query]
[2022-10-20T09:05:00,784][DEBUG][o.e.a.s.TransportSearchAction] [elasticsearch-1] All shards failed for phase: [query]
[2022-10-20T09:05:00,799][DEBUG][o.e.a.s.TransportSearchAction] [elasticsearch-1] All shards failed for phase: [query]
[2022-10-20T09:05:00,812][DEBUG][o.e.a.s.TransportSearchAction] [elasticsearch-1] All shards failed for phase: [query]
[2022-10-20T09:05:00,823][DEBUG][o.e.a.s.TransportSearchAction] [elasticsearch-1] All shards failed for phase: [query]
[2022-10-20T09:05:00,835][DEBUG][o.e.a.s.TransportSearchAction] [elasticsearch-1] All shards failed for phase: [query]
[2022-10-20T09:05:01,314][DEBUG][o.e.a.s.TransportSearchAction] [elasticsearch-1] All shards failed for phase: [query]
[2022-10-20T09:05:01,330][DEBUG][o.e.a.s.TransportSearchAction] [elasticsearch-1] All shards failed for phase: [query]
[2022-10-20T09:05:01,342][DEBUG][o.e.a.s.TransportSearchAction] [elasticsearch-1] All shards failed for phase: [query]
[2022-10-20T09:05:01,358][DEBUG][o.e.a.s.TransportSearchAction] [elasticsearch-1] All shards failed for phase: [query]
[2022-10-20T09:05:01,369][DEBUG][o.e.a.s.TransportSearchAction] [elasticsearch-1] All shards failed for phase: [query]
[2022-10-20T09:05:01,385][DEBUG][o.e.a.s.TransportSearchAction] [elasticsearch-1] All shards failed for phase: [query]
[2022-10-20T09:05:02,396][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-1}{sZD70v3nSgK9y6MsfFvB2A}{6huHryUASNS8_z561jCgEA}{*.*.*.*}{*.*.*.*:9305}{ml.machine_memory=33505226752, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, clusterStateVersion=10}]], butneeded [2]), pinging again
[2022-10-20T09:05:02,412][WARN ][o.e.c.NodeConnectionsService] [elasticsearch-1] failed to connect to node {elasticsearch-2}{eRxN_IvaSfSQ82zeaKw0LQ}{YELeTWyVS3u1h-hWXB0FrA}{*.*.*.*}{*.*.*.*:9305}{ml.machine_memory=33505226752, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} (tried [1] times)
org.elasticsearch.transport.ConnectTransportException: [elasticsearch-2][*.*.*.*:9305] connect_exception
at org.elasticsearch.transport.TcpChannel.awaitConnected(TcpChannel.java:165) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:454) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:117) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.transport.ConnectionManager.internalOpenConnection(ConnectionManager.java:237) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.transport.ConnectionManager.connectToNode(ConnectionManager.java:119) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:369) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:356) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:153) [elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.cluster.NodeConnectionsService$ConnectionChecker.doRun(NodeConnectionsService.java:180) [elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723) [elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.5.1.jar:6.5.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_342]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_342]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_342]
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: *.*.*.*/*.*.*.*:9305
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) ~[?:?]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) ~[?:?]
... 1 more
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) ~[?:?]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) ~[?:?]
... 1 more
[2022-10-20T09:05:05,211][INFO ][o.e.x.m.j.p.NativeController] [elasticsearch-1] Native controller process has stopped - no new native processes can be started
[2022-10-20T09:05:05,212][INFO ][o.e.n.Node ] [elasticsearch-1] stopping ...
[2022-10-20T09:05:05,232][WARN ][o.e.d.z.ZenDiscovery ] [elasticsearch-1] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-1}{sZD70v3nSgK9y6MsfFvB2A}{6huHryUASNS8_z561jCgEA}{*.*.*.*}{*.*.*.*:9305}{ml.machine_memory=33505226752, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, clusterStateVersion=10}]], butneeded [2]), pinging again
[2022-10-20T09:05:05,232][INFO ][o.e.x.w.WatcherService ] [elasticsearch-1] stopping watch service, reason [shutdown initiated]
[2022-10-20T09:05:05,518][INFO ][o.e.n.Node ] [elasticsearch-1] stopped
[2022-10-20T09:05:05,518][INFO ][o.e.n.Node ] [elasticsearch-1] closing ...
[2022-10-20T09:05:05,528][INFO ][o.e.n.Node ] [elasticsearch-1] closed