# Subtle scoring issue with multi-value fields' fieldNorm being calculated as if they are one concatenated value **URL:** \ **Category:** Elasticsearch\ **Created:** [February 20, 2012, 5:56am UTC](https://discuss.elastic.co/t/subtle-scoring-issue-with-multi-value-fields-fieldnorm-being-calculated-as-if-they-are-one-concatenated-value/6752 "2012-02-20T05:56:19Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Tal\_Rotbart](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tal_rotbart/32/2651_2.png) [@Tal\_Rotbart](https://discuss.elastic.co/u/Tal_Rotbart)\ **Post date:** [February 20, 2012, 5:56am UTC](https://discuss.elastic.co/t/subtle-scoring-issue-with-multi-value-fields-fieldnorm-being-calculated-as-if-they-are-one-concatenated-value/6752/1 "2012-02-20T05:56:19Z")
Hi all, I'm facing a subtle scoring issue, which is no doubt Lucene related but I'm wondering if there's a good ES solution to it. A similar problem is summed up for 'that other Lucene search engine' in this well put StackOverflow question: > -- but is still relevant to ES. Essentially, when scoring a multi-valued field -- the length for the fieldNorm is calculated as if the multiple values of the field are concatenated together, instead of having a unique fieldNorm for each value. I'm guessing behind the scene Lucene is indexing the multiple-values as one big value. This penalises multi-value fields significantly. A good example of why this is undesirable is in the above StackOverflow question. Indexing people with multiple aliases: Person 1: David Bowie, David Robert Jones, Ziggy Stardust, Thin White Duke Person 2: David Letterman Person 3: David Hasselhoff, David Michael Hasselhoff Currently, searching for "David" Person 2 comes first and Person 1 comes in last. Intuitively, I'd expect the opposite-- that searching for "David" would bring up Person 1 or Person 3 first (for matching 2 values in the field), and then the other of the two, and Person 2 should come in dead last. It seems Person 1 is penalised for having multiple field values which mean that the field length is increased greatly as the aggregated field value has the most tokens, and the fieldNorm suffers. Is there some ES mapping option to prevent this from happening? Or alternatively, a query DSL directive to prevent it? The alternative workaround is to index each permutation as a separate document and somehow group them in the query -- what is a good way of doing this with ES? Many thanks, Tal ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [February 20, 2012, 6:24am UTC](https://discuss.elastic.co/t/subtle-scoring-issue-with-multi-value-fields-fieldnorm-being-calculated-as-if-they-are-one-concatenated-value/6752/2 "2012-02-20T06:24:43Z")
On Mon, 2012-02-20 at 16:56 +1100, Tal Rotbart wrote: > Hi all, > > I'm facing a subtle scoring issue, which is no doubt Lucene related > but I'm wondering if there's a good ES solution to it. > > A similar problem is summed up for 'that other Lucene search engine' > in this well put StackOverflow question: > [lucene - Scoring of solr multivalued field - Stack Overflow](http://stackoverflow.com/questions/9261524/scoring-of-solr-multivalued-field) > -- but is still relevant to ES. > > Essentially, when scoring a multi-valued field -- the length for the > fieldNorm is calculated as if the multiple values of the field are > concatenated together, instead of having a unique fieldNorm for each > value. Great question, and an interesting link provided by Simon: [http://lucene.apache.org/core/old\_versioned\_docs/versions/3\_0\_3/api/contrib-misc/org/apache/lucene/misc/SweetSpotSimilarity.html](http://lucene.apache.org/core/old_versioned_docs/versions/3_0_3/api/contrib-misc/org/apache/lucene/misc/SweetSpotSimilarity.html) Looking forward to hearing more about this clint ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [February 20, 2012, 6:37am UTC](https://discuss.elastic.co/t/subtle-scoring-issue-with-multi-value-fields-fieldnorm-being-calculated-as-if-they-are-one-concatenated-value/6752/3 "2012-02-20T06:37:06Z")
On Mon, 2012-02-20 at 07:24 +0100, Clinton Gormley wrote: > On Mon, 2012-02-20 at 16:56 +1100, Tal Rotbart wrote: > > > Hi all, > > > > I'm facing a subtle scoring issue, which is no doubt Lucene related > > but I'm wondering if there's a good ES solution to it. > > > > A similar problem is summed up for 'that other Lucene search engine' > > in this well put StackOverflow question: > > [lucene - Scoring of solr multivalued field - Stack Overflow](http://stackoverflow.com/questions/9261524/scoring-of-solr-multivalued-field) > > -- but is still relevant to ES. > > > > Essentially, when scoring a multi-valued field -- the length for the > > fieldNorm is calculated as if the multiple values of the field are > > concatenated together, instead of having a unique fieldNorm for each > > value. Setting the mapping for the field to {"omit\_norms": true} seems to work: > However, I'm not entirely sure exactly what the impact of that mapping is, and why Simon didn't mention it in his response to that question: > Interested to hear more from Lucene knowledgables. clint ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [February 20, 2012, 12:54pm UTC](https://discuss.elastic.co/t/subtle-scoring-issue-with-multi-value-fields-fieldnorm-being-calculated-as-if-they-are-one-concatenated-value/6752/4 "2012-02-20T12:54:52Z")
Yes, omitting norms would do the trick as well, and it makes more sense to set it in this case. On Monday, February 20, 2012 at 8:37 AM, Clinton Gormley wrote: > On Mon, 2012-02-20 at 07:24 +0100, Clinton Gormley wrote: > > > On Mon, 2012-02-20 at 16:56 +1100, Tal Rotbart wrote: > > > > > Hi all, > > > > > > I'm facing a subtle scoring issue, which is no doubt Lucene related > > > but I'm wondering if there's a good ES solution to it. > > > > > > A similar problem is summed up for 'that other Lucene search engine' > > > in this well put StackOverflow question: > > > [lucene - Scoring of solr multivalued field - Stack Overflow](http://stackoverflow.com/questions/9261524/scoring-of-solr-multivalued-field) > > > -- but is still relevant to ES. > > > > > > Essentially, when scoring a multi-valued field -- the length for the > > > fieldNorm is calculated as if the multiple values of the field are > > > concatenated together, instead of having a unique fieldNorm for each > > > value. > > Setting the mapping for the field to {"omit\_norms": true} seems to work: > > [gist:1868139 · GitHub](https://gist.github.com/1868139) > > However, I'm not entirely sure exactly what the impact of that mapping > is, and why Simon didn't mention it in his response to that question: > [lucene - Scoring of solr multivalued field - Stack Overflow](http://stackoverflow.com/a/9283201/819598) > > Interested to hear more from Lucene knowledgables. > > clint ---
**Author:** ![Tal\_Rotbart](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tal_rotbart/32/2651_2.png) [@Tal\_Rotbart](https://discuss.elastic.co/u/Tal_Rotbart)\ **Post date:** [February 20, 2012, 10:16pm UTC](https://discuss.elastic.co/t/subtle-scoring-issue-with-multi-value-fields-fieldnorm-being-calculated-as-if-they-are-one-concatenated-value/6752/5 "2012-02-20T22:16:44Z")
I'll give it a shot and report back. Thanks guys! On 20 February 2012 23:54, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > Yes, omitting norms would do the trick as well, and it makes more sense to > set it in this case. > > On Monday, February 20, 2012 at 8:37 AM, Clinton Gormley wrote: > > On Mon, 2012-02-20 at 07:24 +0100, Clinton Gormley wrote: > > On Mon, 2012-02-20 at 16:56 +1100, Tal Rotbart wrote: > > Hi all, > > I'm facing a subtle scoring issue, which is no doubt Lucene related > but I'm wondering if there's a good ES solution to it. > > A similar problem is summed up for 'that other Lucene search engine' > in this well put StackOverflow question: > [lucene - Scoring of solr multivalued field - Stack Overflow](http://stackoverflow.com/questions/9261524/scoring-of-solr-multivalued-field) > -- but is still relevant to ES. > > Essentially, when scoring a multi-valued field -- the length for the > fieldNorm is calculated as if the multiple values of the field are > concatenated together, instead of having a unique fieldNorm for each > value. > > Setting the mapping for the field to {"omit\_norms": true} seems to work: > > [gist:1868139 · GitHub](https://gist.github.com/1868139) > > However, I'm not entirely sure exactly what the impact of that mapping > is, and why Simon didn't mention it in his response to that question: > [lucene - Scoring of solr multivalued field - Stack Overflow](http://stackoverflow.com/a/9283201/819598) > > Interested to hear more from Lucene knowledgables. > > clint ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:38am UTC](https://discuss.elastic.co/t/subtle-scoring-issue-with-multi-value-fields-fieldnorm-being-calculated-as-if-they-are-one-concatenated-value/6752/6 "2017-07-06T03:38:45Z")