Hey @AdarshPrabhakara ,
- How is the 10,000 candidates chosen?
num_candidates is the same idea as efSearch. Its the number of candidates we continue to keep track of while searching the HNSW graph per shard. This number is applied per shard.
- If we have 1M vector documents, to search across all of these should we pick like 100 shards, so that each shard has max 10k documents? We need very pretty good recall on the retrieved results.
I would say not. 1M vectors should fit in a single shard. HNSW is really good at providing high recall even in larger graphs.
So how do we pick shard sizes when we have this limit of 10k candidates per shards for KNN?
I would say you shouldn't.
If you are wanting 100% recall, then you probably don't want to index the vectors at all and just use brute-force. But keep in mind this scales linearly, where HNSW scales logarithmically and provides much faster query speeds.