Implementing Hybrid Search with k-NN and BM25 in Elasticsearch Open Source

Elastic Stack Elasticsearch
1

Hi,

I am currently using Elasticsearch version 8.16 and am looking to implement hybrid search that combines k-NN and BM25. Since I am using the open-source version, I am unable to utilize RRF. Is there a way to use retrievers with the open-source version? If not, what would be the best alternatives for implementing hybrid search in the open-source version?

Thank you.

2

Welcome!

You can use boosting:

GET ecommerce/_search
{
  "query" : {
    "bool" : {
      "must" : [{
        "match": {
          "description": {
            "query": "summer clothes",
            "boost": 0.1
          }
        }
      },{
        "knn": {
          "field": "desc_embbeding",
          "query_vector": [0.123, 0.244,...],
          "boost": 2.0,
          "filter": {
            "term": {
              "department": "women"
            }
          }
        }
      }],
      "filter" : {
        "range" : { "price": { "lte": 30 } }
      }
    }
  }
}

If you want to do something like RRF, you can run a multi search query.
Then use the different results to compute on your side the RRF "score":

score = 0.0
for q in queries:
    if d in result(q):
        score += 1.0 / ( k + rank( result(q), d ) )
return score

# where
# k is a ranking constant
# q is a query in the set of queries
# d is a document in the result set of q
# result(q) is the result set of q
# rank( result(q), d ) is d's rank within the result(q) starting from 1

Of course, if you are lucky to have an enterprise license, all that will be much more easy to implement...

1 Like
3

Thanks a lot, @dadoonet ! Just to clarify, does this mean there is no way to internally combine BM25 and k-NN results in the open-source version without relying on multi-search(Executes several searches)?

I tried writing a combined script, but it seems there is no way to separately access the k-NN and BM25 scores. From my understanding, the default combination simply adds the k-NN and BM25 scores together.