I am trying to add remote node as a gateway/seed for cross cluster search. I am able to add one cluster but for the other I get connect timeout exception. I ran Transport Tracer on all the involved nodes.
Cross Cluster Node:
[2018-07-27T15:06:00,464][TRACE][o.e.t.T.tracer ] [B4NKXOs] [8][internal:transport/handshake] sent to [{edi-ost#10.60.129.219:9300}{CQGV6NcZQBih_r9XyuVISw}{10.60.129.219}{10.60.129.219:9300}] (timeout: [30s])
[2018-07-27T15:06:00,464][TRACE][o.e.t.T.tracer ] [B4NKXOs] [7][internal:transport/handshake] sent to [{edi-vmaas#10.59.8.89:9300}{YoyiJTMqRS2__zXIOS7gKA}{10.59.8.89}{10.59.8.89:9300}] (timeout: [30s])
[2018-07-27T15:06:00,768][TRACE][o.e.t.T.tracer ] [B4NKXOs] [8][internal:transport/handshake] received response from [{edi-ost#10.60.129.219:9300}{CQGV6NcZQBih_r9XyuVISw}{10.60.129.219}{10.60.129.219:9300}]
[2018-07-27T15:06:00,768][TRACE][o.e.t.T.tracer ] [B4NKXOs] [7][internal:transport/handshake] received response from [{edi-vmaas#10.59.8.89:9300}{YoyiJTMqRS2__zXIOS7gKA}{10.59.8.89}{10.59.8.89:9300}]
[2018-07-27T15:06:01,031][TRACE][o.e.t.T.tracer ] [B4NKXOs] [9][internal:tcp/handshake] sent to [{test-node1}{E24OFhqzRj26KJZel6x9eA}{qGOZ8TTPSkuikDmkbeobrQ}{10.59.8.89}{10.59.8.89:9300}] (timeout: [null])
[2018-07-27T15:06:01,532][TRACE][o.e.t.T.tracer ] [B4NKXOs] [10][internal:transport/handshake] sent to [{test-node1}{E24OFhqzRj26KJZel6x9eA}{qGOZ8TTPSkuikDmkbeobrQ}{10.59.8.89}{10.59.8.89:9300}] (timeout: [30s])
[2018-07-27T15:06:01,791][TRACE][o.e.t.T.tracer ] [B4NKXOs] [10][internal:transport/handshake] received response from [{test-node1}{E24OFhqzRj26KJZel6x9eA}{qGOZ8TTPSkuikDmkbeobrQ}{10.59.8.89}{10.59.8.89:9300}]
[2018-07-27T15:06:31,042][TRACE][o.e.t.T.tracer ] [B4NKXOs] [13][internal:tcp/handshake] sent to [{edi-ost#10.60.149.116:9300}{AjIJDdNKQRyMd3q4MDB-cg}{10.60.149.116}{10.60.149.116:9300}] (timeout: [null])
[2018-07-27T15:06:31,530][TRACE][o.e.t.T.tracer ] [B4NKXOs] [14][internal:transport/handshake] sent to [{edi-ost#10.60.149.116:9300}{AjIJDdNKQRyMd3q4MDB-cg}{10.60.149.116}{10.60.149.116:9300}] (timeout: [30s])
[2018-07-27T15:06:31,791][TRACE][o.e.t.T.tracer ] [B4NKXOs] [14][internal:transport/handshake] received response from [{edi-ost#10.60.149.116:9300}{AjIJDdNKQRyMd3q4MDB-cg}{10.60.149.116}{10.60.149.116:9300}]
[2018-07-27T15:07:01,799][WARN ][o.e.t.RemoteClusterService] [B4NKXOs] failed to update seed list for cluster: edi-ost
org.elasticsearch.transport.ConnectTransportException: [ost-node-1][192.168.192.9:9300] connect_timeout[30s]
at org.elasticsearch.transport.TcpChannel.awaitConnected(TcpChannel.java:163) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:631) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:530) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:331) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler.lambda$collectRemoteNodes$2(RemoteClusterConnection.java:462) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.common.util.CancellableThreads.executeIO(CancellableThreads.java:105) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler.collectRemoteNodes(RemoteClusterConnection.java:444) [elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler.collectRemoteNodes(RemoteClusterConnection.java:504) [elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler$1.doRun(RemoteClusterConnection.java:432) [elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.3.2.jar:6.3.2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_141]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_141]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:626) [elasticsearch-6.3.2.jar:6.3.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_141]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]
EDI-VMaas Cluster (successfull connection)
[2018-07-27T04:36:00,234][TRACE][o.e.t.T.tracer ] [test-node1] [5][internal:tcp/handshake] received request
[2018-07-27T04:36:00,234][TRACE][o.e.t.T.tracer ] [test-node1] [5][internal:tcp/handshake] sent response
[2018-07-27T04:36:00,569][TRACE][o.e.t.T.tracer ] [test-node1] [7][internal:transport/handshake] received request
[2018-07-27T04:36:00,569][TRACE][o.e.t.T.tracer ] [test-node1] [7][internal:transport/handshake] sent response
[2018-07-27T04:36:01,364][TRACE][o.e.t.T.tracer ] [test-node1] [9][internal:tcp/handshake] received request
[2018-07-27T04:36:01,364][TRACE][o.e.t.T.tracer ] [test-node1] [9][internal:tcp/handshake] sent response
[2018-07-27T04:36:01,638][TRACE][o.e.t.T.tracer ] [test-node1] [10][internal:transport/handshake] received request
[2018-07-27T04:36:01,639][TRACE][o.e.t.T.tracer ] [test-node1] [10][internal:transport/handshake] sent response
EDI-OST Cluster has two nodes
Node-1
[2018-07-27T09:36:31,378][TRACE][o.e.t.T.tracer ] [ost-node-1] [13][internal:tcp/handshake] received request
[2018-07-27T09:36:31,378][TRACE][o.e.t.T.tracer ] [ost-node-1] [13][internal:tcp/handshake] sent response
[2018-07-27T09:36:31,635][TRACE][o.e.t.T.tracer ] [ost-node-1] [14][internal:transport/handshake] received request
[2018-07-27T09:36:31,635][TRACE][o.e.t.T.tracer ] [ost-node-1] [14][internal:transport/handshake] sent response
Node-2
[2018-07-27T09:36:00,244][TRACE][o.e.t.T.tracer ] [ost-node-2] [6][internal:tcp/handshake] received request
[2018-07-27T09:36:00,245][TRACE][o.e.t.T.tracer ] [ost-node-2] [6][internal:tcp/handshake] sent response
[2018-07-27T09:36:00,579][TRACE][o.e.t.T.tracer ] [ost-node-2] [8][internal:transport/handshake] received request
[2018-07-27T09:36:00,580][TRACE][o.e.t.T.tracer ] [ost-node-2] [8][internal:transport/handshake] sent response
Could someone help? The cross-cluster nodes and the two OST nodes are running 6.3.2.