Cross Cluster Search param search.remote.initial_connect_timeout does not work

Hi, I am running with cross cluster search in elastic.
I want to raise the search.remote.initial_connect_timeout from the default 30sec to a larger number,
but i see this has no impact on anything.
I would expect it to wait on connection according to configuration, but all i see in the log is:
Caused by: java.util.concurrent.ExecutionException: ConnectTransportException[[....] connect_timeout[30s]]; nested: AnnotatedConnectException[Connection refused: /.....]; nested: ConnectException[Connection refused];

To clarify what i am testing:
I stopped the remote elastic and started to raise it after the local cluster started, and i would expect the local cluster to wait the timeout i configured, but an exception is thrown right away with the 30 seconds timeout message.

Any ideas?
thanks

  1. What version of Elasticsearch are you using?
  2. The connection timeout only impacts how long Elasticsearch will wait for a connection attempt to complete. If the connection attempt fails (for reasons such as connection refused), Elasticsearch will not retry.
  3. It does sound weird that you encountered a situation where the connection was refused, but the exception message mentioned the connection timeout. Are you able to post the complete stack trace?

Hi, i am using elastic 5.6.

Here is the stack trace:

[2019-08-11T01:46:40,172][ERROR][o.e.b.Bootstrap ] [es_node1] Exception
java.lang.IllegalStateException: failed to connect to remote clusters
at org.elasticsearch.transport.RemoteClusterService.initializeRemoteClusters(RemoteClusterService.java:330) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.transport.TransportService.doStart(TransportService.java:227) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:69) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.node.Node.start(Node.java:694) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:278) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:351) [elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132) [elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:123) [elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:70) [elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) [elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) [elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) [elasticsearch-5.6.3.jar:5.6.3]
Caused by: java.util.concurrent.ExecutionException: ConnectTransportException[[:9300] connect_timeout[30s]]; nested: AnnotatedConnectException[Connection refused: /:9300]; nested: ConnectException[Connection refused];
at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.getValue(BaseFuture.java:262) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:235) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:67) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.transport.RemoteClusterService.initializeRemoteClusters(RemoteClusterService.java:324) ~[elasticsearch-5.6.3.jar:5.6.3]
... 12 more
Caused by: org.elasticsearch.transport.ConnectTransportException: [:9300] connect_timeout[30s]
at org.elasticsearch.transport.netty4.Netty4Transport.connectToChannels(Netty4Transport.java:362) ~[?:?]
at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:570) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:117) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.transport.TransportService.openConnection(TransportService.java:351) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler.lambda$collectRemoteNodes$2(RemoteClusterConnection.java:412) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.common.util.CancellableThreads.executeIO(CancellableThreads.java:105) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler.collectRemoteNodes(RemoteClusterConnection.java:409) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler$1.doRun(RemoteClusterConnection.java:396) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.6.3.jar:5.6.3]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_66]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_66]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) ~[elasticsearch-5.6.3.jar:5.6.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_66]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_66]
at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_66]
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /:9300
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_66]
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_66]
[2019-08-11T01:46:40,169][WARN ][o.e.t.RemoteClusterService] [es_node1] failed to update seed list for cluster: cluster_two