# Weird scoring when using multi word synonyms

**URL:** <https://discuss.elastic.co/t/weird-scoring-when-using-multi-word-synonyms/153636>\
**Category:** Elasticsearch\
**Created:** [October 23, 2018, 3:06pm UTC](https://discuss.elastic.co/t/weird-scoring-when-using-multi-word-synonyms/153636 "2018-10-23T15:06:20Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![byronvoorbach](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/byronvoorbach/32/8283_2.png) [@byronvoorbach](https://discuss.elastic.co/u/byronvoorbach)\
**Post date:** [October 23, 2018, 3:06pm UTC](https://discuss.elastic.co/t/weird-scoring-when-using-multi-word-synonyms/153636/1 "2018-10-23T15:06:20Z")

</div>

Hi all!

I just ran into an issue with the scoring of multiword synonyms.  
When a synonym resolves to a multiword synonym, the underlying scoring model seems to change. Resulting in unfair scoring for documents that contain the multi-word synonym.  
I created a small setup to reproduce my issue (bear with me, it's a lot of code):

```
DELETE test
PUT test
{
  "settings": {
    "number_of_replicas": 0,
    "number_of_shards": 1,
    "analysis": {
      "filter": {
        "synonyms": {
          "type": "synonym_graph",
          "synonyms": [
            "usa, america",
            "usa, united states"
          ]
        }
      },
      "analyzer": {
        "synonym_analyzer": {
          "tokenizer": "whitespace",
          "filter": [
            "lowercase",
            "synonyms"
          ]
        }
      }
    }
  },
  "mappings": {
    "test": {
      "properties": {
        "title": {
          "type": "text",
          "analyzer": "synonym_analyzer"
        },
        "description": {
          "type": "text",
          "analyzer": "synonym_analyzer"
        }
      }
    }
  }
}

```

When analyzing two queries, the Lucene queries are as follows:

Result 1:

```
GET test/_validate/query?rewrite=true
{
  "query": {
    "multi_match": {
      "query": "usa",
      "type": "cross_fields", 
      "fields": [
        "title",
        "description"
      ]
    }
  }
}

```

Result 2:

```
GET test/_validate/query?rewrite=true
{
  "query": {
    "multi_match": {
      "query": "america",
      "type": "cross_fields", 
      "fields": [
        "title",
        "description"
      ]
    }
  }
}

```

Result 1:

```
(title:america | description:america) (title:"united states" | description:"united states") (title:usa | description:usa)

```

Result 2:

```
(title:usa | title:america | description:usa | description:america)

```

When executing the above mentioned ES queries and analyzing the result with `explain`, we can see the following effect:

- In the scores of the first query, the individual query parts get SUM'ed.
- In the second query the MAX of the results is used for scoring.

This generates a totally different search result in a big data set.

Results showing this:

```
POST test/test/1
{
  "title": "The USA is a very big country",
  "description": "Book about the states"
}

POST test/test/2
{
  "title": "The USA is a very big country",
  "description": "Movie about the united states"
}
POST test/_refresh

```

Query:

```
GET test/_search
{
  "explain": true, 
  "query": {
    "multi_match": {
      "query": "usa",
      "type": "cross_fields", 
      "fields": [
        "title",
        "description"
      ]
    }
  }
}

```

Explain:

```
 "_source": {
          "title": "The USA is a very big country",
          "description": "Movie about the united states"
        },
 "_explanation": {
  "value": 1.272605,
  "description": "sum of:",
  "details": [
    {
      "value": 0.19856805,
      "description": "max of:",
      "details": [
        {
          "value": 0.19856805,
          "description": "weight(title:america in 1) [PerFieldSimilarity], result of:",
          "details": [
            {
           ... rest removed

```

Query:

```
GET test/_search
{
  "explain": true, 
  "query": {
    "multi_match": {
      "query": "america",
      "type": "cross_fields", 
      "fields": [
        "title",
        "description"
      ]
    }
  }
}

```

Explain:

```
"_source": {
          "title": "The USA is a very big country",
          "description": "Book about the states"
        },
"_explanation": {
  "value": 0.19856805,
  "description": "max of:",
  "details": [
    {
      "value": 0.19856805,
      "description": "weight(title:usa in 0) [PerFieldSimilarity], result of:",
      "details": [
        {
           ... rest removed

```

Is this expected behavior? Or is Lucene/Elasticsearch weird in handling multiword synonym expansions?  
Running Elasticsearch 6.4.2

---

<div class="post-metadata">

**Author:** ![byronvoorbach](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/byronvoorbach/32/8283_2.png) [@byronvoorbach](https://discuss.elastic.co/u/byronvoorbach)\
**Post date:** [November 14, 2018, 2:23pm UTC](https://discuss.elastic.co/t/weird-scoring-when-using-multi-word-synonyms/153636/2 "2018-11-14T14:23:12Z")

</div>

Bump 😊

---

<div class="post-metadata">

**Author:** ![Jaspreet\_Singh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jaspreet_singh/32/58296_2.png) [@Jaspreet\_Singh](https://discuss.elastic.co/u/Jaspreet_Singh)\
**Post date:** [November 14, 2018, 9:25pm UTC](https://discuss.elastic.co/t/weird-scoring-when-using-multi-word-synonyms/153636/3 "2018-11-14T21:25:51Z")

</div>

When querying with `usa` your query gets translated into `usa, united, states and america` (matches both fields). And when querying with `america`, you only get `usa and america`. (matches only `title` field). This is what impacts the overall scores. What you see is whatever fields gets applied to the score. If there is no score for a particular match, you will not see it under `explain`. Doesnt really mean the scoring structure or scheme is different.

---

<div class="post-metadata">

**Author:** ![softwaredoug](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/softwaredoug/32/22681_2.png) [@softwaredoug](https://discuss.elastic.co/u/softwaredoug)\
**Post date:** [November 14, 2018, 11:25pm UTC](https://discuss.elastic.co/t/weird-scoring-when-using-multi-word-synonyms/153636/4 "2018-11-14T23:25:15Z")

</div>

Scoring is impacted by a couple of things, and its better to look at the validate/explain without rewrite on:

```auto
GET test/_validate/query?explain=true
{
  "query": {
    "multi_match": {
      "query": "usa",
      "type": "cross_fields", 
      "fields": [
        "title",
        "description"
      ]}}}

```

Which gives

```auto
blended(terms:[title:usa, title:america, description:usa, description:america])

```

and

```auto
(blended(terms:[title:america, description:america]) (title:\"united states\" | description:\"united states\") blended(terms:[title:usa, description:usa]))

```

Which is what I would expect. Scoring is impacted by:

- Blending takes each term and blends the document frequency, to compute the same idf for each term
- Blending then takes the _dismax_ of each term score, so the highesc scoring term
- Blending does not support phrase queries, so in the latter case the query cannot be fully blended and instead there's a dismax of each representation of the full phrase
- Phrase queries have their own document frequency scoring - approximating the idf by taking the sum of the constituent term's idfs

Shameless plug, but we go through this in our [relevance training](http://opensourceconnections.com/events/training) and I'll give you a sneak peak at this slide which I think is helpful:

![image](https://us1.discourse-cdn.com/elastic/original/3X/2/5/2526e047576b05d54f196fa0574b36a2bf28f35a.png)

---

<div class="post-metadata">

**Author:** ![abdon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abdon/32/9195_2.png) [@abdon](https://discuss.elastic.co/u/abdon)\
**Post date:** [November 15, 2018, 8:28am UTC](https://discuss.elastic.co/t/weird-scoring-when-using-multi-word-synonyms/153636/5 "2018-11-15T08:28:09Z")

</div>

I would like to add one more thing: you are using the `synonym_graph` token filter at both index and query time. Generally it is not necessary to expand synonyms at both index and query time. If you are already indexing all synonyms, there is no need to also search for all those synonyms.

In fact, the `synonyms_graph` token filter is designed to be used at query time only (see [this blog](https://www.elastic.co/blog/multitoken-synonyms-and-graph-queries-in-elasticsearch) for more details). Using it at index time may lead to unexpected results.

I'd recommend that you use the `synonym_analyzer` as the [`search_analyzer`](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-analyzer.html) in your mapping for the `title` and `description` fields, instead of as the `analyzer`.

---

<div class="post-metadata">

**Author:** ![byronvoorbach](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/byronvoorbach/32/8283_2.png) [@byronvoorbach](https://discuss.elastic.co/u/byronvoorbach)\
**Post date:** [November 15, 2018, 8:53am UTC](https://discuss.elastic.co/t/weird-scoring-when-using-multi-word-synonyms/153636/6 "2018-11-15T08:53:17Z")

</div>

Think that was a mis-copy when I made the snippet, results are actually generated with normal synonym filter.

---

<div class="post-metadata">

**Author:** ![byronvoorbach](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/byronvoorbach/32/8283_2.png) [@byronvoorbach](https://discuss.elastic.co/u/byronvoorbach)\
**Post date:** [November 15, 2018, 8:59am UTC](https://discuss.elastic.co/t/weird-scoring-when-using-multi-word-synonyms/153636/7 "2018-11-15T08:59:33Z")

</div>

Thanks for your feedback Doug,

Think I understand a bit more what's going on and what to do to combat this.  
Would be nice to have more control in the way synonyms are scored.

Easiest way at the moment is by rewriting all multiword synonyms into a single term to make sure scoring works the same for all terms

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 13, 2018, 8:59am UTC](https://discuss.elastic.co/t/weird-scoring-when-using-multi-word-synonyms/153636/8 "2018-12-13T08:59:52Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.