# Storage issues in es vectorized retrieval

**URL:** <https://discuss.elastic.co/t/storage-issues-in-es-vectorized-retrieval/368343>\
**Category:** Elasticsearch\
**Created:** [October 7, 2024, 11:27am UTC](https://discuss.elastic.co/t/storage-issues-in-es-vectorized-retrieval/368343 "2024-10-07T11:27:53Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![chao\_xi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chao_xi/32/138139_2.png) [@chao\_xi](https://discuss.elastic.co/u/chao_xi)\
**Post date:** [October 7, 2024, 11:27am UTC](https://discuss.elastic.co/t/storage-issues-in-es-vectorized-retrieval/368343/1 "2024-10-07T11:27:53Z")

</div>

When I searched for similarity again, I found the following problem. I matched the content with the highest similarity displayed by es's vector cosine similarity. I manually compared the question vector with the vector of the content with the highest similarity retrieved by es, and the question vector with the vector of the content that I thought should have the highest similarity. I found that the vector I thought was higher was indeed more similar.

```auto
dense_vector_query = {
            "size": top_k,
            "query": {
                "script_score": {
                    "query": {
                        "match_all": {} 
                    },
                    "script": {
                        "source": "cosineSimilarity(params.query_vector, 'embedding') + 1.0",
                        "params": {
                            "query_vector": embeddings  
                        }
                    }
                }
            }
        }

```

---

<div class="post-metadata">

**Author:** ![Alex\_Salgado-Elastic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_salgado-elastic/32/103081_2.png) [@Alex\_Salgado-Elastic](https://discuss.elastic.co/u/Alex_Salgado-Elastic)\
**Post date:** [October 8, 2024, 12:00am UTC](https://discuss.elastic.co/t/storage-issues-in-es-vectorized-retrieval/368343/2 "2024-10-08T00:00:07Z")

</div>

Hi @chao_xi , welcome to our community.

How many dimensions are you defining? It's not mandatory but, you could ensure your vector embeddings are normalized if they are not. You can normalize them either before indexing them.

Example:

```auto
def normalize(vector):
    norm = np.linalg.norm(vector)
    return vector / norm if norm > 0 else vector

```

---

<div class="post-metadata">

**Author:** ![chao\_xi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chao_xi/32/138139_2.png) [@chao\_xi](https://discuss.elastic.co/u/chao_xi)\
**Post date:** [October 8, 2024, 12:19am UTC](https://discuss.elastic.co/t/storage-issues-in-es-vectorized-retrieval/368343/3 "2024-10-08T00:19:05Z")

</div>

Thank you for your reply! As for the vector, I used the embeddings model of BGE, which converts the text into a 1024-dimensional vector. I directly used this vector for retrieval. I don't quite understand what you mean by normalization.

---

<div class="post-metadata">

**Author:** ![chao\_xi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chao_xi/32/138139_2.png) [@chao\_xi](https://discuss.elastic.co/u/chao_xi)\
**Post date:** [October 8, 2024, 12:26am UTC](https://discuss.elastic.co/t/storage-issues-in-es-vectorized-retrieval/368343/4 "2024-10-08T00:26:35Z")

</div>

I compare the code for similarity

```auto
def calculate_similarity(vector1, vector2):
        dot_product = np.dot(vector1, vector2)
        magnitude1 = np.linalg.norm(vector1)
        magnitude2 = np.linalg.norm(vector2)
        similarity = dot_product / (magnitude1 * magnitude2)
        return similarity

```

---

<div class="post-metadata">

**Author:** ![sunyicode0012](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sunyicode0012/32/138161_2.png) [@sunyicode0012](https://discuss.elastic.co/u/sunyicode0012)\
**Post date:** [October 8, 2024, 8:19am UTC](https://discuss.elastic.co/t/storage-issues-in-es-vectorized-retrieval/368343/5 "2024-10-08T08:19:47Z")

</div>

Hi folks, I'm also encountering this issue, looking forward to a solution.