We use separate data and client nodes in our cluster, all reads go through the client nodes. So client nodes act as co-ordinating nodes in our case. Now for this given setup I have couple of questions in my mind,
What happens to the requests in the search queue of the data nodes if the client node that had forwarded those requests goes out of the cluster due to OOM ?
Will those request still get executed, even when the client node (also the co-ordinating node) is not there to collect and merge the outputs from the data nodes.
Will some other client node take the place of the client node that went OOM and do the merge.
In the second case, what happens if the data node that had the request forwarded from a client node goes OOM.
Will the client node detect that the data node is not responding and send the request to some other data node having a replica of this shard ?
Will it infinitely wait for the data node that has gone OOM to recover back & return result ? or will it collect the result from other data nodes (ignoring this node that went OOM) and merge the result.
I'm assuming an environment where there are multiple shards, with two replicas.