Difference between KNN similarity and document score

Elastic Search
1

Assume we have some documents with a VECTOR field storing normalized vectors in it. this is the mapping we use for this field:

{
  type: 'dense_vector',
  similarity: 'dot_product',
  index: true,
}

Now, we perform KNN on this field:

{
      field: `FIELD`,
      k,
      num_candidates,
      query_vector,
      similarity: minimumSimilarity,
      filter: {
        bool: {
          must: elasticMust,
          must_not: itemId ? [termTranslator('itemId', itemId)] : [],
        },
      },
    }

The problem is that the returned documents does not have any indicator of the similarity. they have score but when we set the minimumSimilarity as 0.3, we don't get documents with the score of 0.6.

I used the profile feature and found that there is a relation between similarity and document score:

VectorSimilarityQuery[similarity=0.3, docScore=0.65, innerKnnQuery=DocAndScore[10000]]

VectorSimilarityQuery[similarity=0.4, docScore=0.7, innerKnnQuery=DocAndScore[10000]]

VectorSimilarityQuery[similarity=0.812, docScore=0.906, innerKnnQuery=DocAndScore[10000]]

We want to give user the similarity to let them set minimumSimilarity based on it.

2

You're correct that similarity is used to calculate the final score, but it's not accessible in the way you're looking for. We do have some information on how scores are calculated if that's of interest - but I don't think you really want to go down the route of filtering on this type of metric.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.