Hi. Given millions of documents in our index, we would like to run ANN in order to get the top-X documents, where X is greater than 10,000. However, we're running into the
num_candidates cannot exceed 10000 error.
We don't mind making multiple calls if needed to get more than 10k results. I've experimented with using
search_after together with ANN and
num_candidates=10000 - but this simply returns 0 results after the first 10k.
I know we can replace ANN with a brute-force exact-kNN, but we would prefer to use ANN for the performance benefits if possible.
Is there any way at all to use ANN to get more than 10k results?