I've moved recently to use cross cluster search to connect my Kibana to two separate clusters instead of using tribe nodes. I am finding certain cases where trying to query from indexes present in only of of the clusters causes the gateway node to reset and my cluster state turns yellow until the node has recovered. All elastic components are version 5.6.2.
If I query the indexes using tribe nodes I have no such problem
My clusters ids are "aws" and "ovh" and the indexes are only in the "aws" cluster
I have tried using index patterns *:log-cloudflare-* or aws:log-cloudflare-* and in both cases I get the following error:
Error: Request to Elasticsearch failed: {"error":{"root_cause":[{"type":"transport_serialization_exception","reason":"Failed to deserialize response of type [org.elasticsearch.search.fetch.QueryFetchSearchResult]"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"log-cloudflare-aws-2018.02.21","node":"TOGXlOADSgqMmgFzb568Eg","reason":{"type":"transport_serialization_exception","reason":"Failed to deserialize response of type [org.elasticsearch.search.fetch.QueryFetchSearchResult]","caused_by":{"type":"index_out_of_bounds_exception","reason":"readerIndex(48) + length(1) exceeds writerIndex(48): UnpooledDuplicatedByteBuf(ridx: 48, widx: 48, cap: 65536, unwrapped: PooledHeapByteBuf(ridx: 6, widx: 48, cap: 65536))"}}}]},"status":500}
KbnError@http://kibana.ovh.xcl.ie/bundles/commons.bundle.js?v=15543:83:18358
RequestFailure@http://kibana.ovh.xcl.ie/bundles/commons.bundle.js?v=15543:83:19107
callResponseHandlers/<@http://kibana.ovh.xcl.ie/bundles/kibana.bundle.js?v=15543:29:12495
[337]/</Promise.try@http://kibana.ovh.xcl.ie/bundles/commons.bundle.js?v=15543:82:22229
[337]/</Promise.map/<@http://kibana.ovh.xcl.ie/bundles/commons.bundle.js?v=15543:82:21588
[337]/</Promise.map@http://kibana.ovh.xcl.ie/bundles/commons.bundle.js?v=15543:82:21553
callResponseHandlers@http://kibana.ovh.xcl.ie/bundles/kibana.bundle.js?v=15543:29:12103
fetchWithStrategy/<@http://kibana.ovh.xcl.ie/bundles/kibana.bundle.js?v=15543:29:366
processQueue@http://kibana.ovh.xcl.ie/bundles/commons.bundle.js?v=15543:38:23621
scheduleProcessQueue/<@http://kibana.ovh.xcl.ie/bundles/commons.bundle.js?v=15543:38:23888
$eval@http://kibana.ovh.xcl.ie/bundles/commons.bundle.js?v=15543:39:4607
$digest@http://kibana.ovh.xcl.ie/bundles/commons.bundle.js?v=15543:39:2343
$apply@http://kibana.ovh.xcl.ie/bundles/commons.bundle.js?v=15543:39:5026
done@http://kibana.ovh.xcl.ie/bundles/commons.bundle.js?v=15543:37:25016
completeRequest@http://kibana.ovh.xcl.ie/bundles/commons.bundle.js?v=15543:37:28702
createHttpBackend/</xhr.onload@http://kibana.ovh.xcl.ie/bundles/commons.bundle.js?v=15543:37:29634