Does k-nearest neighbor (kNN) search - supports search_after pagination?

Elastic Stack Elasticsearch
1

Reference:

  1. kNN search: k-nearest neighbor (kNN) search | Elasticsearch Guide [master] | Elastic
  2. Search after pagination: https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html#search-aft
1 Like
2

I have a same question. What is the suggested pagination approach if using KNN?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

4

Sorry for the slow response! Yes, search_after is supported with approximate kNN search. You can think of the 'knn' section as adding 'k' matching documents to the search. If you set k=100 and size=20, then you will only show the first 20 matching documents. Then you could use search_after (or another paging technique like from/ size) to page through the rest of the matching documents.

However, with approximate kNN search it is not possible to page through all documents in order of their vector similarity. For example, you cannot say "return only the nearest 'k' documents, now fetch the next nearest 'k' documents, etc." until you page through all the documents. Approximate kNN search algorithms do not allow this sort of paging.

If you need to do that, you either need to set a large 'k' up-front like 'k=100', so you can page through those 'k' documents. Or, you could use exact kNN (k-nearest neighbor (kNN) search | Elasticsearch Guide [8.5] | Elastic) if you really need to be able to page through the whole set of documents in order of vector similarity.