I got cross cluster search working but after some time it simply breaks without any log entry (neither on the local cluster nor on the remote one). It happens the following:
- The remote cluster is already running since days.
- I start the local cluster and the cross cluster search simply works. Reliable and fast.
- I wait some time (1 or 2 hours) and perform again the same cross cluster search and it simply hangs until the timeout is reached. Then the following message is printed on the local cluster logfiles:
[2017-11-10T14:04:29,127][WARN ][r.suppressed ] path: /:/_search, params: {q=*, index=:}
org.elasticsearch.transport.TransportException: unable to communicate with remote cluster []
at org.elasticsearch.transport.RemoteClusterService$1.onFailure(RemoteClusterService.java:244) ~[elasticsearch-5.6.4.jar:5.6.4]
at org.elasticsearch.transport.RemoteClusterConnection$1.handleException(RemoteClusterConnection.java:188) ~[elasticsearch-5.6.4.jar:5.6.4]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1077) ~[elasticsearch-5.6.4.jar:5.6.4]
at org.elasticsearch.transport.TransportService$Adapter.lambda$onConnectionClosed$6(TransportService.java:903) ~[elasticsearch-5.6.4.jar:5.6.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-5.6.4.jar:5.6.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
Caused by: org.elasticsearch.transport.NodeDisconnectedException: [][:9300][indices:admin/shards/search_shards] disconnected
Any clue of what can be going on?
Many thanks in advance!