Data nodes are connected to the master but not to each other during upgrade

I am doing ES upgrade from 6.8.2 to 7.9.3 and seeing issues with 2 data nodes. Shard movements are going from "index" to "init" stage and logs show that the data nodes are not connected to each other.
On the new node I see NodeNotConnectedException:

org.elasticsearch.transport.NodeNotConnectedException: [xx.xx.xx.xx:9300] Node not connected
        at org.elasticsearch.transport.ClusterConnectionManager.getConnection( ~[elasticsearch-7.9.3.jar:7.9.3]
        at org.elasticsearch.transport.TransportService.getConnection( ~[elasticsearch-7.9.3.jar:7.9.3]
        at ~[elasticsearch-7.9.3.jar:7.9.3]
        at$buildConnectionLookup$6( ~[elasticsearch-7.9.3.jar:7.9.3]
        at ~[elasticsearch-7.9.3.jar:7.9.3]
        at ~[elasticsearch-7.9.3.jar:7.9.3]

While on old data nodes, I see this:

[2021-07-12T22:31:53,209][WARN ][o.e.t.OutboundHandler    ] [xxxxxx-xxxxxx] send message failed [channel: Netty4TcpChannel{localAddress=xx.xx.xx.xx/xx.xx.xx.xx:9300, remoteAddress=/xx.xx.xx.xx:34692}]
java.nio.channels.ClosedChannelException: null
	at$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.