Data nodes are connected to the master but not to each other during upgrade

[163satish (163)]

I am doing ES upgrade from 6.8.2 to 7.9.3 and seeing issues with 2 data nodes. Shard movements are going from "index" to "init" stage and logs show that the data nodes are not connected to each other.
On the new node I see NodeNotConnectedException:

org.elasticsearch.transport.NodeNotConnectedException: [xx.xx.xx.xx:9300] Node not connected
        at org.elasticsearch.transport.ClusterConnectionManager.getConnection(ClusterConnectionManager.java:189) ~[elasticsearch-7.9.3.jar:7.9.3]
        at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:673) ~[elasticsearch-7.9.3.jar:7.9.3]
        at org.elasticsearch.action.search.SearchTransportService.getConnection(SearchTransportService.java:373) ~[elasticsearch-7.9.3.jar:7.9.3]
        at org.elasticsearch.action.search.TransportSearchAction.lambda$buildConnectionLookup$6(TransportSearchAction.java:537) ~[elasticsearch-7.9.3.jar:7.9.3]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.getConnection(AbstractSearchAsyncAction.java:587) ~[elasticsearch-7.9.3.jar:7.9.3]
        at org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction.executePhaseOnShard(SearchQueryThenFetchAsyncAction.java:79) ~[elasticsearch-7.9.3.jar:7.9.3]

While on old data nodes, I see this:

[2021-07-12T22:31:53,209][WARN ][o.e.t.OutboundHandler    ] [xxxxxx-xxxxxx] send message failed [channel: Netty4TcpChannel{localAddress=xx.xx.xx.xx/xx.xx.xx.xx:9300, remoteAddress=/xx.xx.xx.xx:34692}]
java.nio.channels.ClosedChannelException: null
	at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]