# Failing to use dis\_max to get accurate scores on multiple fields

**URL:** <https://discuss.elastic.co/t/failing-to-use-dis-max-to-get-accurate-scores-on-multiple-fields/7038>\
**Category:** Elasticsearch\
**Created:** [March 17, 2012, 2:33pm UTC](https://discuss.elastic.co/t/failing-to-use-dis-max-to-get-accurate-scores-on-multiple-fields/7038 "2012-03-17T14:33:53Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![AEvar\_Arnfjord\_Bjarm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aevar_arnfjord_bjarm/32/2746_2.png) [@AEvar\_Arnfjord\_Bjarm](https://discuss.elastic.co/u/AEvar_Arnfjord_Bjarm)\
**Post date:** [March 17, 2012, 2:33pm UTC](https://discuss.elastic.co/t/failing-to-use-dis-max-to-get-accurate-scores-on-multiple-fields/7038/1 "2012-03-17T14:33:53Z")

</div>

I'm having problems getting best name matches when making a query  
across multiple fields with dis\_max, here's a complete test case for  
what I'm doing:

```
https://gist.github.com/2059195

```

In summary I'm trying to search through data like:

doc 1:  
"object\_name" : "Albino Elephant"  
doc 2:  
"object\_name" : "Cute Albino Elephant"  
"object\_name\_other\_language" : "Cute Albino Elephant"  
doc 3:  
"object\_name" : "The Cutest Albino Elephant"  
"object\_name\_other\_language" : "The Cutest Albino Elephant"

I.e. a bunch of objects that have multiple names that have been  
translated into different languages (because I want more accurate  
matches than using an array, and I want special analyzers per  
language).

When I do a search for "Albino Elephant" with this query:

```
    "dis_max" : {
        "queries" : [
            {
                "text" : {
                    "object_name.unmunged" : "Albino Elephant"
                }
            },
            {
                "text" : {
                    "object_name_other_language.unmunged" :

```

"Albino Elephant"  
}  
}  
]  
}

doc #3 is going to be the highest scored hit, but if I were to change  
doc #1 to:

doc 1:  
"object\_name" : "Albino Elephant"  
"object\_name\_other\_language" : "Albino Elephant"

It would be the number #1 hit, so seemingly dis\_max is behaving like a  
bool/should query and aggregating the scores. I thought the whole  
point of dis\_max was to do "execute all these queries, compute their  
scores, and pick the highest one" so you could search through  
e.g. multiple translations and not give objects extra scores by virtue  
of having more translations.

Or maybe something else is up here, Clinton Gormley suggested on IRC  
that this might be because I had \>1 shards, but I changed the gist to  
only create one shard and still have the same results.

---

<div class="post-metadata">

**Author:** ![AEvar\_Arnfjord\_Bjarm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aevar_arnfjord_bjarm/32/2746_2.png) [@AEvar\_Arnfjord\_Bjarm](https://discuss.elastic.co/u/AEvar_Arnfjord_Bjarm)\
**Post date:** [March 22, 2012, 12:14pm UTC](https://discuss.elastic.co/t/failing-to-use-dis-max-to-get-accurate-scores-on-multiple-fields/7038/2 "2012-03-22T12:14:04Z")

</div>

On Sat, Mar 17, 2012 at 15:33, Ævar Arnfjörð Bjarmason [avarab@gmail.com](mailto:avarab@gmail.com) wrote:

> I'm having problems getting best name matches when making a query  
> across multiple fields with dis\_max, here's a complete test case for  
> what I'm doing:
> 
> [ElasticSearch Exact Name matching · GitHub](https://gist.github.com/2059195)
> 
> In summary I'm trying to search through data like:
> 
> doc 1:  
> "object\_name" : "Albino Elephant"  
> doc 2:  
> "object\_name" : "Cute Albino Elephant"  
> "object\_name\_other\_language" : "Cute Albino Elephant"  
> doc 3:  
> "object\_name" : "The Cutest Albino Elephant"  
> "object\_name\_other\_language" : "The Cutest Albino Elephant"
> 
> I.e. a bunch of objects that have multiple names that have been  
> translated into different languages (because I want more accurate  
> matches than using an array, and I want special analyzers per  
> language).
> 
> When I do a search for "Albino Elephant" with this query:
> 
> ```
> "dis_max" : {
> "queries" : [
> {
> "text" : {
> "object_name.unmunged" : "Albino Elephant"
> }
> },
> {
> "text" : {
> "object_name_other_language.unmunged" :
> 
> ```
> 
> "Albino Elephant"  
> }  
> }  
> ]  
> }
> 
> doc #3 is going to be the highest scored hit, but if I were to change  
> doc #1 to:
> 
> doc 1:  
> "object\_name" : "Albino Elephant"  
> "object\_name\_other\_language" : "Albino Elephant"
> 
> It would be the number #1 hit, so seemingly dis\_max is behaving like a  
> bool/should query and aggregating the scores. I thought the whole  
> point of dis\_max was to do "execute all these queries, compute their  
> scores, and pick the highest one" so you could search through  
> e.g. multiple translations and not give objects extra scores by virtue  
> of having more translations.
> 
> Or maybe something else is up here, Clinton Gormley suggested on IRC  
> that this might be because I had \>1 shards, but I changed the gist to  
> only create one shard and still have the same results.

I've played with this some more but I haven't gotten it working.  
Anyone have an idea what might be going on here?

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [March 25, 2012, 11:21am UTC](https://discuss.elastic.co/t/failing-to-use-dis-max-to-get-accurate-scores-on-multiple-fields/7038/3 "2012-03-25T11:21:55Z")

</div>

Heya,

If you set explain to true, you will see part of how the score is  
calculated. You will see that dismax is doing its job, choosing the higher  
ranking query that matched as the final score of the document. But, the  
higher ranking doc is the one on object\_name\_other\_language, and thats  
because of the inverse document frequency mainly, because those terms are  
more unique on the mentioned field than the object\_name.

-shay.banon

On Thu, Mar 22, 2012 at 2:14 PM, Ævar Arnfjörð Bjarmason  
[avarab@gmail.com](mailto:avarab@gmail.com)wrote:

> On Sat, Mar 17, 2012 at 15:33, Ævar Arnfjörð Bjarmason [avarab@gmail.com](mailto:avarab@gmail.com)  
> wrote:
> 
> > I'm having problems getting best name matches when making a query  
> > across multiple fields with dis\_max, here's a complete test case for  
> > what I'm doing:
> > 
> > [ElasticSearch Exact Name matching · GitHub](https://gist.github.com/2059195)
> > 
> > In summary I'm trying to search through data like:
> > 
> > doc 1:  
> > "object\_name" : "Albino Elephant"  
> > doc 2:  
> > "object\_name" : "Cute Albino Elephant"  
> > "object\_name\_other\_language" : "Cute Albino Elephant"  
> > doc 3:  
> > "object\_name" : "The Cutest Albino Elephant"  
> > "object\_name\_other\_language" : "The Cutest Albino Elephant"
> > 
> > I.e. a bunch of objects that have multiple names that have been  
> > translated into different languages (because I want more accurate  
> > matches than using an array, and I want special analyzers per  
> > language).
> > 
> > When I do a search for "Albino Elephant" with this query:
> > 
> > ```
> > "dis_max" : {
> > "queries" : [
> > {
> > "text" : {
> > "object_name.unmunged" : "Albino Elephant"
> > }
> > },
> > {
> > "text" : {
> > "object_name_other_language.unmunged" :
> > 
> > ```
> > 
> > "Albino Elephant"  
> > }  
> > }  
> > ]  
> > }
> > 
> > doc #3 is going to be the highest scored hit, but if I were to change  
> > doc #1 to:
> > 
> > doc 1:  
> > "object\_name" : "Albino Elephant"  
> > "object\_name\_other\_language" : "Albino Elephant"
> > 
> > It would be the number #1 hit, so seemingly dis\_max is behaving like a  
> > bool/should query and aggregating the scores. I thought the whole  
> > point of dis\_max was to do "execute all these queries, compute their  
> > scores, and pick the highest one" so you could search through  
> > e.g. multiple translations and not give objects extra scores by virtue  
> > of having more translations.
> > 
> > Or maybe something else is up here, Clinton Gormley suggested on IRC  
> > that this might be because I had \>1 shards, but I changed the gist to  
> > only create one shard and still have the same results.
> 
> I've played with this some more but I haven't gotten it working.  
> Anyone have an idea what might be going on here?

---

<div class="post-metadata">

**Author:** ![AEvar\_Arnfjord\_Bjarm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aevar_arnfjord_bjarm/32/2746_2.png) [@AEvar\_Arnfjord\_Bjarm](https://discuss.elastic.co/u/AEvar_Arnfjord_Bjarm)\
**Post date:** [March 26, 2012, 4:47pm UTC](https://discuss.elastic.co/t/failing-to-use-dis-max-to-get-accurate-scores-on-multiple-fields/7038/4 "2012-03-26T16:47:15Z")

</div>

On Sun, Mar 25, 2012 at 13:21, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote:

> If you set explain to true, you will see part of how the score is  
> calculated. You will see that dismax is doing its job, choosing the higher  
> ranking query that matched as the final score of the document. But, the  
> higher ranking doc is the one on object\_name\_other\_language, and thats  
> because of the inverse document frequency mainly, because those terms are  
> more unique on the mentioned field than the object\_name.

Ah, is there some way to work around that? I see why that's happening  
but that means in a document with a field per language the ones that  
happen to have translations in relatively rare languages will be  
bumped up more than other documents.

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [March 27, 2012, 1:18pm UTC](https://discuss.elastic.co/t/failing-to-use-dis-max-to-get-accurate-scores-on-multiple-fields/7038/5 "2012-03-27T13:18:40Z")

</div>

Maybe boost the regular name one?

On Mon, Mar 26, 2012 at 6:47 PM, Ævar Arnfjörð Bjarmason  
[avarab@gmail.com](mailto:avarab@gmail.com)wrote:

> On Sun, Mar 25, 2012 at 13:21, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote:
> 
> > If you set explain to true, you will see part of how the score is  
> > calculated. You will see that dismax is doing its job, choosing the  
> > higher  
> > ranking query that matched as the final score of the document. But, the  
> > higher ranking doc is the one on object\_name\_other\_language, and thats  
> > because of the inverse document frequency mainly, because those terms are  
> > more unique on the mentioned field than the object\_name.
> 
> Ah, is there some way to work around that? I see why that's happening  
> but that means in a document with a field per language the ones that  
> happen to have translations in relatively rare languages will be  
> bumped up more than other documents.

---

<div class="post-metadata">

**Author:** ![AEvar\_Arnfjord\_Bjarm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aevar_arnfjord_bjarm/32/2746_2.png) [@AEvar\_Arnfjord\_Bjarm](https://discuss.elastic.co/u/AEvar_Arnfjord_Bjarm)\
**Post date:** [March 28, 2012, 9:47am UTC](https://discuss.elastic.co/t/failing-to-use-dis-max-to-get-accurate-scores-on-multiple-fields/7038/6 "2012-03-28T09:47:12Z")

</div>

On Tue, Mar 27, 2012 at 15:18, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote:

> Maybe boost the regular name one?

The issue is that I don't know in advance what the regular name is,  
since the actual use case is that I'm searching through a bunch of  
languages, so proper boost values would depend on how they're  
distributed throghout the various documents.

What I'm really trying to do is this:

- Search through documents with fields like these:

- The reason I don't structure it like this instead:

One way to work around this I guess would be to just declare that  
English is the default language, and then if I _don't_ have a  
translation for a given language slot just insert the English  
one. This would result in a lot of redundant data being added to the  
index, but might alleviate the IDF problem.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 3:34am UTC](https://discuss.elastic.co/t/failing-to-use-dis-max-to-get-accurate-scores-on-multiple-fields/7038/7 "2017-07-06T03:34:27Z")

</div>