Cross Cluster Search param search.remote.initial_connect_timeout does not work

michal1 · August 4, 2019, 11:26am

Hi, I am running with cross cluster search in elastic.
I want to raise the search.remote.initial_connect_timeout from the default 30sec to a larger number,
but i see this has no impact on anything.
I would expect it to wait on connection according to configuration, but all i see in the log is:
Caused by: java.util.concurrent.ExecutionException: ConnectTransportException[[....] connect_timeout[30s]]; nested: AnnotatedConnectException[Connection refused: /.....]; nested: ConnectException[Connection refused];

To clarify what i am testing:
I stopped the remote elastic and started to raise it after the local cluster started, and i would expect the local cluster to wait the timeout i configured, but an exception is thrown right away with the 30 seconds timeout message.

Any ideas?
thanks

Tim_Brooks · August 10, 2019, 12:43am

What version of Elasticsearch are you using?
The connection timeout only impacts how long Elasticsearch will wait for a connection attempt to complete. If the connection attempt fails (for reasons such as connection refused), Elasticsearch will not retry.
It does sound weird that you encountered a situation where the connection was refused, but the exception message mentioned the connection timeout. Are you able to post the complete stack trace?

michal1 · August 11, 2019, 6:23am

Hi, i am using elastic 5.6.

Here is the stack trace:

[2019-08-11T01:46:40,172][ERROR][o.e.b.Bootstrap ] [es_node1] Exception
java.lang.IllegalStateException: failed to connect to remote clusters
at org.elasticsearch.transport.RemoteClusterService.initializeRemoteClusters(RemoteClusterService.java:330) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.transport.TransportService.doStart(TransportService.java:227) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:69) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.node.Node.start(Node.java:694) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:278) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:351) [elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132) [elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:123) [elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:70) [elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) [elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) [elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) [elasticsearch-5.6.3.jar:5.6.3]
Caused by: java.util.concurrent.ExecutionException: ConnectTransportException[[:9300] connect_timeout[30s]]; nested: AnnotatedConnectException[Connection refused: /:9300]; nested: ConnectException[Connection refused];
at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.getValue(BaseFuture.java:262) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:235) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:67) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.transport.RemoteClusterService.initializeRemoteClusters(RemoteClusterService.java:324) ~[elasticsearch-5.6.3.jar:5.6.3]
... 12 more
Caused by: org.elasticsearch.transport.ConnectTransportException: [:9300] connect_timeout[30s]
at org.elasticsearch.transport.netty4.Netty4Transport.connectToChannels(Netty4Transport.java:362) ~[?:?]
at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:570) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:117) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.transport.TransportService.openConnection(TransportService.java:351) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler.lambda$collectRemoteNodes$2(RemoteClusterConnection.java:412) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.common.util.CancellableThreads.executeIO(CancellableThreads.java:105) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler.collectRemoteNodes(RemoteClusterConnection.java:409) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler$1.doRun(RemoteClusterConnection.java:396) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.6.3.jar:5.6.3]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_66]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_66]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) ~[elasticsearch-5.6.3.jar:5.6.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_66]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_66]
at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_66]
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /:9300
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_66]
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_66]
[2019-08-11T01:46:40,169][WARN ][o.e.t.RemoteClusterService] [es_node1] failed to update seed list for cluster: cluster_two

system · September 8, 2019, 6:23am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.