# Filter vector search results to get only relevant documents? **URL:** \ **Category:** Elasticsearch\ **Tags:** vector-search\ **Created:** [July 28, 2023, 11:57am UTC](https://discuss.elastic.co/t/filter-vector-search-results-to-get-only-relevant-documents/339543 "2023-07-28T11:57:13Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![john\_nicolas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/john_nicolas/32/123249_2.png) [@john\_nicolas](https://discuss.elastic.co/u/john_nicolas)\ **Post date:** [July 28, 2023, 11:57am UTC](https://discuss.elastic.co/t/filter-vector-search-results-to-get-only-relevant-documents/339543/1 "2023-07-28T11:57:13Z")
I am not an expert in elastic queries, I have not found a solution to filter my results. My index contains 450,000 documents. The issue is that when I perform a search, it always returns all 450,000 documents, sorted by relevance. However, upon examining the last results, some of them do not even match my query. Therefore, I am considering the idea of limiting and filtering the results only for the documents with a similarity score greater than 0.5. This way, we can ensure that we will only get relevant results. ```auto mappings of field : "vector": { "type": "dense_vector", "dims": 768, "index": false }, ``` my query like : ```auto query["bool"]["should"].append({ "script_score": { "query": {"match_all": {}}, "script": { "source": "(1.0 + cosineSimilarity(params['query_vector'], 'vector'))", "params": {"query_vector": vector_field} } } }) ``` i want to get only docs that have : ` cosineSimilarity(params['query_vector'], 'vector') > 0.5` ---
**Author:** ![BenTrent](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bentrent/32/33915_2.png) [@BenTrent](https://discuss.elastic.co/u/BenTrent)\ **Post date:** [July 28, 2023, 12:16pm UTC](https://discuss.elastic.co/t/filter-vector-search-results-to-get-only-relevant-documents/339543/2 "2023-07-28T12:16:02Z")
Heya @john_nicolas , A thing to try would be using the `min_score` parameter: [Search API | Elasticsearch Guide [8.8] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/8.8/search-search.html#search-api-min-score) This requires documents to have at least this score if they are going to be included in the result set. ---
**Author:** ![john\_nicolas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/john_nicolas/32/123249_2.png) [@john\_nicolas](https://discuss.elastic.co/u/john_nicolas)\ **Post date:** [July 28, 2023, 12:34pm UTC](https://discuss.elastic.co/t/filter-vector-search-results-to-get-only-relevant-documents/339543/3 "2023-07-28T12:34:29Z")
ok i tired it but it does'nt work or i dont know how we can use it the problem is that i use a combined query that combines a simple search with the vector search like this : ```auto query["bool"]["should"].append({ "multi_match": { "query": value, "fields": ["title_txt_ml", "content_txt_ml"], "operator": "and", "boost": 80.0 } }) query["bool"]["should"].append({ "multi_match": { "query": value, "fields": ["title_txt_ml", "content_txt_ml"], "operator": "and", "type": "phrase", "slop": 1, "boost": 20.0 } }) query["bool"]["should"].append({ "script_score": { "query": {"match_all": {}}, "script": { "source": "(1.0 + cosineSimilarity(params['query_vector'], 'vector'))", "params": {"query_vector": vector_field} }, "boost":0.5 } }) ``` so i connot define a min\_score for pertinence of docs , i think that the solution is to filter results of only vector search , i.e in the bloc ```auto query["bool"]["should"].append({ "script_score": { "query": {"match_all": {}}, "script": { "source": "(1.0 + cosineSimilarity(params['query_vector'], 'vector'))", "params": {"query_vector": vector_field} }, "boost":0.5 } }) ``` ---
**Author:** ![BenTrent](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bentrent/32/33915_2.png) [@BenTrent](https://discuss.elastic.co/u/BenTrent)\ **Post date:** [July 28, 2023, 4:44pm UTC](https://discuss.elastic.co/t/filter-vector-search-results-to-get-only-relevant-documents/339543/4 "2023-07-28T16:44:08Z")
Ah, you only want `min_score` for vectors but continue to get the docs for other matching queries. In newer versions `knn` has this capability: [k-nearest neighbor (kNN) search | Elasticsearch Guide [8.9] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/knn-search.html#_search_knn_with_expected_similarity) using calculated `similarity` of the document. But, I will have to spend some time thinking on how to do this with brute-force knn. We want to make all these interactions cleaner, the API will improve :). ---
**Author:** ![BenTrent](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bentrent/32/33915_2.png) [@BenTrent](https://discuss.elastic.co/u/BenTrent)\ **Post date:** [July 28, 2023, 5:21pm UTC](https://discuss.elastic.co/t/filter-vector-search-results-to-get-only-relevant-documents/339543/5 "2023-07-28T17:21:18Z")
Hey @john_nicolas you mentioned you tried `min_score`, but did you try `min_score` within the `script_score`? ```auto { "script_score": { "query": {"match_all": {}}, "script": { "source": "(1.0 + cosineSimilarity(params['query_vector'], 'vector'))", "params": {"query_vector": vector_field} }, "min_score": 0.5, "boost":0.5 } } ``` ---
**Author:** ![Mark\_Harwood1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood1/32/101255_2.png) [@Mark\_Harwood1](https://discuss.elastic.co/u/Mark_Harwood1)\ **Post date:** [July 29, 2023, 7:54am UTC](https://discuss.elastic.co/t/filter-vector-search-results-to-get-only-relevant-documents/339543/6 "2023-07-29T07:54:12Z")
Something that may be of help in deciding what score threshold to pick: If you have many category fields (eg department:kitchen utensils) you can use these to see what score ranges produce a bewildering number of values (indicating random rather than cohesive set of concepts) In [this visualisation](https://twitter.com/elasticmark/status/1126874836180008960?s=46&t=v-3AtimwGhnBtq6NDs5Pew) query score is on the x-axis and the breakdown of document categories matching that score band is in the vertical bar. The high-scoring documents come from a small selection of related categories while the lower scoring documents come from a huge number of unrelated categories. In a way the number of categories provide a measure of how many different meanings the matches in a range have and give a reasonable indication of where there are meaningless results in the score bands. ---
**Author:** ![john\_nicolas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/john_nicolas/32/123249_2.png) [@john\_nicolas](https://discuss.elastic.co/u/john_nicolas)\ **Post date:** [July 29, 2023, 3:58pm UTC](https://discuss.elastic.co/t/filter-vector-search-results-to-get-only-relevant-documents/339543/8 "2023-07-29T15:58:59Z")
exactly i want min\_score only for vectors but continue get the docs for others matching queries i change my method : ` > "knnvector": { "type": "elastiknn\_dense\_float\_vector", "elastiknn": { "dims": 768, "model": "lsh", "similarity": "angular", "L": 99, "k": 1 } } ` , i tried this to put min\_score in the query but it is not work for me ```auto payload['query'] = { "elastiknn_nearest_neighbors": { "field": "knnvector", "vec": { "values": vector_field, }, "model": "lsh", "similarity": "angular", "candidates": 5000, "k": 5, "min_score":1.5 } } ``` ---
**Author:** ![BenTrent](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bentrent/32/33915_2.png) [@BenTrent](https://discuss.elastic.co/u/BenTrent)\ **Post date:** [July 29, 2023, 4:36pm UTC](https://discuss.elastic.co/t/filter-vector-search-results-to-get-only-relevant-documents/339543/9 "2023-07-29T16:36:18Z")
`elastiknn_dense_float_vector` isn't officially supported by Elasticsearch, and I am not even sure its currently maintained. If you want approximate nearest neighbors, you should try `dense_vector` with `index: true`. That allows you to set expected similarity thresholds at query time. Or you can continue to use `script_query` and set `min_score` for that specific query. ---
**Author:** ![john\_nicolas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/john_nicolas/32/123249_2.png) [@john\_nicolas](https://discuss.elastic.co/u/john_nicolas)\ **Post date:** [July 29, 2023, 4:47pm UTC](https://discuss.elastic.co/t/filter-vector-search-results-to-get-only-relevant-documents/339543/10 "2023-07-29T16:47:24Z")
it does'nt work the min\_score for the specificc query i tried this but not work for me : ```auto "script_score": { "query": {"match_all": {}}, "script": { "source": "(1.0 + cosineSimilarity(params['query_vector'], 'vector'))", "params": {"query_vector": vector_field} }, "min_score": 1.5, } ``` ---
**Author:** ![john\_nicolas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/john_nicolas/32/123249_2.png) [@john\_nicolas](https://discuss.elastic.co/u/john_nicolas)\ **Post date:** [July 29, 2023, 5:12pm UTC](https://discuss.elastic.co/t/filter-vector-search-results-to-get-only-relevant-documents/339543/11 "2023-07-29T17:12:00Z")
ok so finaly i used : ```auto "esvector": { "type": "dense_vector", "dims": 768, "index": true, "similarity": "cosine" }, ``` i tried this query : ```auto payload['query'] =my_complex_queries payload['knn'] ={ "field": "esvector", "query_vector": vector_field, "candidates":100, "k":5, "similarity":0.5 } ``` but does'nt work , we can pass similarity threshold like this ? ---
**Author:** ![BenTrent](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bentrent/32/33915_2.png) [@BenTrent](https://discuss.elastic.co/u/BenTrent)\ **Post date:** [August 2, 2023, 12:15pm UTC](https://discuss.elastic.co/t/filter-vector-search-results-to-get-only-relevant-documents/339543/12 "2023-08-02T12:15:55Z")
@john_nicolas Could you clarify "but does'nt work" ? Are you getting errors? Unexpected irrelevant results? Could you also let us know your Elasticsearch version? I am surprised that the `min_score` within the script\_query didn't work either. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 30, 2023, 12:16pm UTC](https://discuss.elastic.co/t/filter-vector-search-results-to-get-only-relevant-documents/339543/13 "2023-08-30T12:16:09Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.