I'm trying to setup a cross cluster search. When I try to add a remote node as a seed, I'm getting a connect_timeout.
Error in cross cluster node:
[2018-07-25T23:08:51,936][WARN ][o.e.t.RemoteClusterService] [qR5I4sH] failed to update seed list for cluster: edi-ost-test
org.elasticsearch.transport.ConnectTransportException: [test-node-1][192.168.192.9:9300] connect_timeout[30s]
at org.elasticsearch.transport.netty4.Netty4Transport.connectToChannels(Netty4Transport.java:363) ~[?:?]
at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:570) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:473) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:342) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler.lambda$collectRemoteNodes$2(RemoteClusterConnection.java:431) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.common.util.CancellableThreads.executeIO(CancellableThreads.java:105) ~[elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler.collectRemoteNodes(RemoteClusterConnection.java:415) [elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler$1.doRun(RemoteClusterConnection.java:402) [elasticsearch-5.6.10.jar:5.6.10]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.10.jar:5.6.10]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_141]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_141]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.10.jar:5.6.10]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_141]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]
Caused by: io.netty.channel.ConnectTimeoutException: connection timed out: 192.168.192.9/192.168.192.9:9300
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:267) ~[?:?]
at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38) ~[?:?]
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:120) ~[?:?]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
... 1 more
I added a Transport Tracer in my remote node and it is logging the following whenever my cross cluster node starts up:
[2018-07-25T17:38:21,579][TRACE][o.e.t.T.tracer ] [test-node-1] [6][internal:tcp/handshake] received request
[2018-07-25T17:38:21,580][TRACE][o.e.t.T.tracer ] [test-node-1] [6][internal:tcp/handshake] sent response
[2018-07-25T17:38:21,831][TRACE][o.e.t.T.tracer ] [test-node-1] [9][internal:transport/handshake] received request
[2018-07-25T17:38:21,831][TRACE][o.e.t.T.tracer ] [test-node-1] [9][internal:transport/handshake] sent response
Both ES are 5.6.10. Any idea whats happening?