# How to calculate score by dividing number of occurences per field length?

**URL:** <https://discuss.elastic.co/t/how-to-calculate-score-by-dividing-number-of-occurences-per-field-length/212625>\
**Category:** Elasticsearch\
**Tags:** painless\
**Created:** [December 20, 2019, 8:13am UTC](https://discuss.elastic.co/t/how-to-calculate-score-by-dividing-number-of-occurences-per-field-length/212625 "2019-12-20T08:13:09Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Slawomir\_Balcerzak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/slawomir_balcerzak/32/47746_2.png) [@Slawomir\_Balcerzak](https://discuss.elastic.co/u/Slawomir_Balcerzak)\
**Post date:** [December 20, 2019, 8:13am UTC](https://discuss.elastic.co/t/how-to-calculate-score-by-dividing-number-of-occurences-per-field-length/212625/1 "2019-12-20T08:13:09Z")

</div>

I would like to implement my own scoring algorithm using script\_score. It should to take count of occurrences of searched words and divide this number by field length. But I don't know how to gather those two variables from Elasticsearch engine at the same time. I tried to do something like this on mapping side:

```
"settings": {
        "similarity": {
            "my_similarity": {
                "type": "scripted",
                "script": {
                    "source": "return doc.freq/doc.length;"
                }
            },
}

```

but the problem is that doc.length is not the length of a field. I also tried to calculate it on query side:

```
"query": {
    "function_score": {
        "script_score": {
            "script": {
                "lang": "painless",
                "inline": "freq/doc['some_field'].length;"
            }
        }
    }
}

```

but here I can't get freq variable. Is there any way to compute it somehow?

Maybe it is possible to return doc.freq as a defined field by some mapping configuration, and calculate it on query side?

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [December 20, 2019, 9:06am UTC](https://discuss.elastic.co/t/how-to-calculate-score-by-dividing-number-of-occurences-per-field-length/212625/2 "2019-12-20T09:06:54Z")

</div>

You may want to check out the [scripted similarity](https://www.elastic.co/guide/en/elasticsearch/reference/7.5/index-modules-similarity.html#scripted_similarity) instead of a function score query in order to implement this.

---

<div class="post-metadata">

**Author:** ![Slawomir\_Balcerzak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/slawomir_balcerzak/32/47746_2.png) [@Slawomir\_Balcerzak](https://discuss.elastic.co/u/Slawomir_Balcerzak)\
**Post date:** [December 20, 2019, 9:30am UTC](https://discuss.elastic.co/t/how-to-calculate-score-by-dividing-number-of-occurences-per-field-length/212625/3 "2019-12-20T09:30:40Z")

</div>

Yes, I checked it already out, but I think I'm not able to achieve what I want using scripted similarity API. As I mentioned in my previous post the problem is `doc.length` property, which seems to be not a field length. This single property returns different values for the same document field when I use different queries.

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [December 20, 2019, 10:10am UTC](https://discuss.elastic.co/t/how-to-calculate-score-by-dividing-number-of-occurences-per-field-length/212625/4 "2019-12-20T10:10:30Z")

</div>

have you used `explain: true` in the search request to check for different doc.length? can you provide a fully reproducible example with the similarity, index creation, mapping and the queries returning different length (which might be due to the mapping)?

---

<div class="post-metadata">

**Author:** ![Slawomir\_Balcerzak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/slawomir_balcerzak/32/47746_2.png) [@Slawomir\_Balcerzak](https://discuss.elastic.co/u/Slawomir_Balcerzak)\
**Post date:** [December 20, 2019, 3:05pm UTC](https://discuss.elastic.co/t/how-to-calculate-score-by-dividing-number-of-occurences-per-field-length/212625/5 "2019-12-20T15:05:14Z")

</div>

This is my mapping, where I override score script with single property `doc.length` :

```auto
    {
        "mappings": {
            "some_info": {
                "_all": {
                    "enabled": "false"
                },
                "properties": {
                    "some_text_field": {
                        "type": "text",
                        "index": "false",
                        "fields": {
                            "ngram": {
                                "type": "text",
                                "analyzer": "custom_nGram_analyzer",
                                "similarity": "my_similarity"
                            }
                        }
                    }
                }
            }
        },
        "settings": {
            "analysis": {
                "analyzer": {
                    "custom_nGram_analyzer": {
                        "type": "custom",
                        "tokenizer": "whitespace",
                        "filter": [
                            "lowercase",
                            "asciifolding",
                            "custom_nGram_filter"
                        ]
                    }
                },
                "filter": {
                    "custom_nGram_filter": {
                        "type": "ngram",
                        "token_chars": [],
                        "min_gram": 3,
                        "max_gram": 16
                    }
                },
                "normalizer": {
                    "custom_lowercase_normalizer": {
                        "type": "custom",
                        "filter": [
                            "lowercase"
                        ]
                    }
                }
            },
            "similarity": {
                "my_similarity": {
                    "type": "scripted",
                    "script": {
                        "source": "return doc.length;"
                    }
                }
            }
        }
    }

```

```
data: POST: /some_index/some_info
{
    "some_text_field": "Lorem Ipsum is simply ...."
}

```

when I perform query: **GET** /some\_index/\_search?q=Lorem  
I get score: **76.0**  
but when I do: **GET** /some\_index/\_search?q=Lorem Ipsum  
I get score: **152.0**

\_explaination shows something like this:

```auto
    "_explanation": {
        "value": 152.0,
        "description": "max of:",
        "details": [
            {
                "value": 152.0,
                "description": "sum of:",
                "details": [
                    {
                        "value": 76.0,
                        "description": "weight(Synonym(some_text_field.ngram:lor some_text_field.ngram:lore some_text_field.ngram:lorem some_text_field.ngram:ore some_text_field.ngram:orem some_text_field.ngram:rem) in 0) [PerFieldSimilarity], result of:",
                        "details": [
                            {
                                "value": 76.0,
                                "description": "score from ScriptedSimilarity(weightScript=[null], script=[Script{type=inline, lang='painless', idOrCode='return doc.length;', options={}, params={}}]) computed from:",
                                "details": [
                                    ...
                                    {
                                        "value": 76.0,
                                        "description": "doc.length",
                                        "details": []
                                    }
                                ]
                            }
                        ]
                    },
                    {
                        "value": 76.0,
                        "description": "weight(Synonym(some_text_field.ngram:ips some_text_field.ngram:ipsu some_text_field.ngram:ipsum some_text_field.ngram:psu some_text_field.ngram:psum some_text_field.ngram:sum) in 0) [PerFieldSimilarity], result of:",
                        "details": [
                            {
                                "value": 76.0,
                                "description": "score from ScriptedSimilarity(weightScript=[null], script=[Script{type=inline, lang='painless', idOrCode='return doc.length;', options={}, params={}}]) computed from:",
                                "details": [
                                        ...
                                    {
                                        "value": 76.0,
                                        "description": "doc.length",
                                        "details": []
                                    }
                                ]
                            }
                        ]
                    }
                ]
            }
        ]

```

Is there any way to get doc.length without summing it?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 17, 2020, 3:05pm UTC](https://discuss.elastic.co/t/how-to-calculate-score-by-dividing-number-of-occurences-per-field-length/212625/6 "2020-01-17T15:05:17Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.