Can we use Approximate kNN algorithm other than Hierarchical Navigable Small World Algorithm?

Elastic Stack Elasticsearch
1

Hi All,

We have to construct a search system which contains 30 millions+ dense vectors.
However, Naive kNN algorithm can not be applied to this scale of data.
And, Hierarchical Navigable Small World Algorithm, which explicitly mentioned as a Approximate k-NN algorithm in a document, is also needs very large memory space.
Therefore, we hope to utilize a Product Quantization Algorithm or other methods for this purpose.

Anyone have any idea?

2

Hello Hiraku,
Currently Elasticsearch supports only precise kNN or HNSW algorithm, product quantization is not supported. But you are welcome submit an issue for request either in Elasticsearch github or Lucene (the library underneath Elasticsearch).