Aggregate Score for Hybrid Search

BenTrent · February 16, 2023, 6:35pm

OK, got a bug fix ready!

github.com/elastic/elasticsearch

Fix explain and hybrid search scoring bug

elastic:main ← benwtrent:bugfix/fix-knn-hybrid-search-behavior-and-explain

opened 06:30PM - 16 Feb 23 UTC

benwtrent

+38 -6

This PR addresses bugs with KNN search. First, when searching with KNN, using… `explain: true` would give strange results. Even though a document obviously matches because of its score, the explanation would indicate that it did NOT match. This bug was because our explanation check was not accounting for the leaf-segment's docBase. Second, when there are multiple segments, and KNN is used in hybrid search, some documents may go unscored by KNN. The main issue here is the disjunction logic within `BoolQuery`. All hybrid queries with `knn` are pure disjunction queries under the hood. And Lucene does its best to optimize the documents it gathers in these types of queries. Consequently, it will look for the maximum possible score for an entire segment leaf. However, the underlying KNN query would ALWAYS increment this passed docId by the segment's docBase (this is fairly ironic given the `explain` bug :D). This would cause the `docId` for calculating the maximum score to roll over to the negatives and thus knn would indicate its maximum score to be `0` for certain segments with higher docBase values. closes: https://github.com/elastic/elasticsearch/issues/93830

Thanks again!

Kok_Gin_Xian · February 17, 2023, 1:12am

Wow, that is quick. When will the bug fix incorporated into a new release? Meanwhile I can work with the workaround first.

Topic		Replies	Views
Distinguishing hybrid search results Elasticsearch vector-search	3	278	October 30, 2023
Elasticsearch Hybrid Query - No Results Elasticsearch	2	840	February 18, 2021
Hybrid Search high score on irrelevant documents Elasticsearch vector-search	9	649	September 23, 2024
Inconsistent hybrid search hits total results lead to incorrect aggregations Elasticsearch vector-search	2	212	November 18, 2024
Why "knn_query" doesn’t have a separate k parameter? Elasticsearch vector-search	13	756	May 9, 2024

Aggregate Score for Hybrid Search

Related topics