# Custom function for Text Similarity Search

**URL:** <https://discuss.elastic.co/t/custom-function-for-text-similarity-search/210158>\
**Category:** Elasticsearch\
**Created:** [December 2, 2019, 10:39am UTC](https://discuss.elastic.co/t/custom-function-for-text-similarity-search/210158 "2019-12-02T10:39:00Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Loreto\_Parisi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/loreto_parisi/32/37830_2.png) [@Loreto\_Parisi](https://discuss.elastic.co/u/Loreto_Parisi)\
**Post date:** [December 2, 2019, 10:39am UTC](https://discuss.elastic.co/t/custom-function-for-text-similarity-search/210158/1 "2019-12-02T10:39:00Z")

</div>

I'm using a `dense_vector` and `cosineSimilarity` to get documents similarity following the good tutorial [here](https://www.elastic.co/blog/text-similarity-search-with-vectors-in-elasticsearch)

```python
script_query = {
        "script_score": {
            "query": {"match_all": {}},
            "script": {
                "source": "cosineSimilarity(params.query_vector, doc['text_vector']) + 1.0",
                "params": {"query_vector": query_vector}
            }
        }
    }

```

As for release 7.3 elasticsearch provide natively for the script the `cosineSimilarity`. When working in the STS (Sentence Textual Similarity), there are more choices for the metrics, among them:

- Euclidean distance
- Manhattan distance
- Cosine distance (equivalente alla Euclidean distance dei vettori normalizzati)
- Hamming distance
- Dot (Inner) Product distance

So, how to implement a custom ElasticSearch similarity function for the search query script, let's say Euclidean or dot product?

Thank you.

**NOTE**

- My reference project was [BertSearch](https://github.com/Hironsan/bertsearch), were the textual embedding has been calculated with Google's BERT.
- We should keep in mind that for a given vectorial distance to get the similarity a common transformation is:`similarity = 1 / (1 + distance)`.
- Regarding BERT, a good similarity scoring approach is described in [bert\_score](https://github.com/Tiiiger/bert_score)

---

<div class="post-metadata">

**Author:** ![mayya](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mayya/32/83147_2.png) [@mayya](https://discuss.elastic.co/u/mayya)\
**Post date:** [December 2, 2019, 8:49pm UTC](https://discuss.elastic.co/t/custom-function-for-text-similarity-search/210158/2 "2019-12-02T20:49:10Z")

</div>

Hello!  
From 7.3 we have the following [vector functions](https://www.elastic.co/guide/en/elasticsearch/reference/7.3/query-dsl-script-score-query.html#vector-functions) available: `cosineSimilarity` and `dotProduct`.

From 7.4 two more [functions](https://www.elastic.co/guide/en/elasticsearch/reference/7.4/query-dsl-script-score-query.html#vector-functions) added: `l1norm` (manhattan distance) and `l2norm` (euclidean distance).

We are still [investigating the need for bit vectors](https://github.com/elastic/elasticsearch/issues/48322) and hamming distance.

> how to implement a custom Elasticsearch similarity function for the search query script, let's say Euclidean or dot product?

They are already implemented from 7.3 (dotProduct) and 7.4 (euclidean distance).  
There is no a straightforward approach to implement custom distance functions, as this would require the development of plugins. If you think some function is widely used and not implemented yet, please open an issue in the elasticsearch github and we will discuss it.

> We should keep in mind that for a given vectorial distance to get the similarity a common transformation is: `similarity = 1 / (1 + distance)` .

In `script_score` query, you can do any transformation with the calculated distance including the one you needed.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 30, 2019, 8:49pm UTC](https://discuss.elastic.co/t/custom-function-for-text-similarity-search/210158/3 "2019-12-30T20:49:14Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.