Aggregate Score for Hybrid Search

BenTrent · February 16, 2023, 6:35pm

OK, got a bug fix ready!

github.com/elastic/elasticsearch

Fix explain and hybrid search scoring bug

elastic:main ← benwtrent:bugfix/fix-knn-hybrid-search-behavior-and-explain

opened 06:30PM - 16 Feb 23 UTC

benwtrent

+38 -6

This PR addresses bugs with KNN search. First, when searching with KNN, using… `explain: true` would give strange results. Even though a document obviously matches because of its score, the explanation would indicate that it did NOT match. This bug was because our explanation check was not accounting for the leaf-segment's docBase. Second, when there are multiple segments, and KNN is used in hybrid search, some documents may go unscored by KNN. The main issue here is the disjunction logic within `BoolQuery`. All hybrid queries with `knn` are pure disjunction queries under the hood. And Lucene does its best to optimize the documents it gathers in these types of queries. Consequently, it will look for the maximum possible score for an entire segment leaf. However, the underlying KNN query would ALWAYS increment this passed docId by the segment's docBase (this is fairly ironic given the `explain` bug :D). This would cause the `docId` for calculating the maximum score to roll over to the negatives and thus knn would indicate its maximum score to be `0` for certain segments with higher docBase values. closes: https://github.com/elastic/elasticsearch/issues/93830

Thanks again!

Kok_Gin_Xian · February 17, 2023, 1:12am

Wow, that is quick. When will the bug fix incorporated into a new release? Meanwhile I can work with the workaround first.

system · March 17, 2023, 1:13am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.