Aggregate Score for Hybrid Search

Elastic Stack Elasticsearch
1

Hi, I'm new to Elasticsearch and am trying out the new hybrid search by specifying the "knn" and "query" parameters in my search.

I set k=100 in knn, size=k=100 in the search request.

For pure vector search (omitting the "query" parameter), I got 100 hits ranked according to the cosine similarity score. This is consistent with my expectations.

However, with hybrid search (adding the "query" parameter with a match-all constant score filter with a boost of 1.0 for example, only the first 25 hits _scores are (1.0 + cosine similarity score), the remaining 75 hits have _scores of 1.0. Changing the "k" parameter in knn does not change this behavior.

query={
            "constant_score": {
                "filter": {
                    "match_all": {}
                },
                "boost": 1.0
            }
        }

knn={
            "field": "image_vector",
            "query_vector": query_vector,
            "k": k,
            "num_candidates": 100,
        }

What am I doing wrong? I'm using Elasticsearch 8.6.1. Please help. I'm using the Python Elasticsearch client v8.6.1 for search.

2

Also, the number of KNN results in the hybrid search hits varies depending on what is searched (the query vector used in KNN). This number is not always the set k value. This condition can be easily seen when there is a "large" gap in the hit _scores when hybrid search is used.

3

Hey @Kok_Gin_Xian ,

These are indeed interesting results. I will try to replicate them myself. Is there a test dataset that replicates your results?

Also, when you don't have the constant_score filter, and you are able to see all the "k" values, are any of those scores negative?

Thanks!

4

Omitting "query" for constant scoring, I am able to get all the k nearest neighbors . I used cosine similarity and the score for all the k neighbors are in the range of 0.60 - 0.67. I didn't see any negative values. My dataset size is sizable 5m so I think that is also why my cosine scores are quite high.

However, adding the "query" for hybrid search, I see the top few hits ~1.6, then a sudden drop to the constant score 1.0. And the number of hits with 1.x are different for different query vector but repeatable for the same query vector.

5

I also tried different versions of Elasticsearch 8.6.0 and 8.5.3 and see no improvement.

6

@Kok_Gin_Xian I am trying to replicate the issue myself and am not having any luck.

What are your number of shards in the index?

Does the index have any deleted documents?

GET _cat/indices?v should give you shard count and number of documents & deleted docs.

Does EVERY document have a vector field? Or are some documents missing the vector field?

To figure this out, you can do an Exists query | Elasticsearch Guide [8.6] | Elastic

7

Hi, this is the output from the request.

'health status index uuid pri rep docs.count docs.deleted store.size pri.store.size\nyellow open image-index AWtVrHStQ3GDJLNoUToAXQ 1 1 3056822 0 29.8gb 29.8gb\n'

9

All data samples should be complete, but let me know if there is a way to check in Elasticsearch.

10

I've retried the inserting of data and can confirm all the fields for the samples are complete, including the vector data.

I also casted the data type of the vector data to float32 and it didn't help with the problem.

11

@BenTrent , wondering if you managed to try my simple image search web app? I don't know how to proceed from here.

12

@Kok_Gin_Xian that doesn't really help me debug. I am still looking into it. I still cannot replicate.

The easiest way to debug is to attempt to create a minimal working replication of the issue.

@Kok_Gin_Xian does the same problem occur if you use the Kibana dev console for your search?

Additionally, on one of the failing documents (the ones that show up as just 1.0 in the middle of the list), can you call the explain API with your hybrid search? Explain API | Elasticsearch Guide [8.6] | Elastic

13

Additionally, on one of the failing documents (the ones that show up as just 1.0 in the middle of the list), can you call the explain API with your hybrid search? Explain API | Elasticsearch Guide [8.6] | Elastic

This is incorrect. Could you make your failing call but with "explain": True in the search body as well? It will output a ton, and may take longer but it will indicate the query clauses that score on that document hit.

14

Ok, let me prepare you a minimal application for reproducing the problem.

15

This is the csv containing the _score and _explanation for the hits.

_score,_explanation
1.6622047,"{'value': 1.6622047, 'description': 'sum of:', 'details': [{'value': 0.66220474, 'description': 'within top k documents', 'details': []}, {'value': 1.0, 'description': 'ConstantScore(*:*)', 'details': []}]}"
1.6613073,"{'value': 1.6613073, 'description': 'sum of:', 'details': [{'value': 0.6613073, 'description': 'within top k documents', 'details': []}, {'value': 1.0, 'description': 'ConstantScore(*:*)', 'details': []}]}"
1.6607261,"{'value': 1.6607261, 'description': 'sum of:', 'details': [{'value': 0.660726, 'description': 'within top k documents', 'details': []}, {'value': 1.0, 'description': 'ConstantScore(*:*)', 'details': []}]}"
1.6599699,"{'value': 1.6599699, 'description': 'sum of:', 'details': [{'value': 0.6599699, 'description': 'within top k documents', 'details': []}, {'value': 1.0, 'description': 'ConstantScore(*:*)', 'details': []}]}"
1.6598208,"{'value': 1.6598208, 'description': 'sum of:', 'details': [{'value': 0.65982085, 'description': 'within top k documents', 'details': []}, {'value': 1.0, 'description': 'ConstantScore(*:*)', 'details': []}]}"
1.0,"{'value': 1.0, 'description': 'sum of:', 'details': [{'value': 1.0, 'description': 'ConstantScore(*:*)', 'details': []}]}"
1.0,"{'value': 1.0, 'description': 'sum of:', 'details': [{'value': 1.0, 'description': 'ConstantScore(*:*)', 'details': []}]}"
1.0,"{'value': 1.0, 'description': 'sum of:', 'details': [{'value': 1.0, 'description': 'ConstantScore(*:*)', 'details': []}]}"
1.0,"{'value': 1.0, 'description': 'sum of:', 'details': [{'value': 1.0, 'description': 'ConstantScore(*:*)', 'details': []}]}"
1.0,"{'value': 1.0, 'description': 'sum of:', 'details': [{'value': 1.0, 'description': 'ConstantScore(*:*)', 'details': []}]}"

This is the src for the search in Python:

text = "lion"

text_embed = encode_text(text)
query_vector = text_embed.tolist()

k = 10

resp = es.search(
    index="image-index",
    size=k,
    request_timeout=30,
    query={"constant_score": {"filter": {"match_all": {}}, "boost": 1.0}},
    knn={
        "field": "image_vector",
        "query_vector": query_vector,
        "k": k,
        "num_candidates": 100,
    },
    _source=["image_url", "image_desc", "filetype"],
    explain=True
)
16

Please see this GitHub repo for a minimal reproducible application.

17

The explanation I get for the hits blew my mind and I don't understand one bit what it is doing.

18

Thank you so much @Kok_Gin_Xian for all the info! I will dig in and report back!

19

@Kok_Gin_Xian I was able to replicate! Thanks!

This behavior is indeed surprising. I will see if I can find the cause.

Here is the bug: Vector search hybrid score surprising behavior · Issue #93830 · elastic/elasticsearch · GitHub

I will be working on figuring out the cause and seeing about a fix.

Thank you so much for the replication data and steps! It makes this all much simpler.

20

I'm glad you are able to replicate the problem. Thank you so much for the help in replicating the problem and working on the bug!

21

Hey @Kok_Gin_Xian

I experimented and a current work around could be force-merging the index to a single segment.

So

POST <index>/_forcemerge?max_num_segments=1

I am still digging into what the exact cause is over multiple segments in the same shard.

Also, I did find a bug with explain that i will be fixing as well.

Thank you for helping make Elasticsearch better!!!