# Score is lower if text is longer **URL:** \ **Category:** Elasticsearch\ **Created:** [November 29, 2011, 11:25am UTC](https://discuss.elastic.co/t/score-is-lower-if-text-is-longer/6007 "2011-11-29T11:25:55Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Robert](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robert/32/3055_2.png) [@Robert](https://discuss.elastic.co/u/Robert)\ **Post date:** [November 29, 2011, 11:25am UTC](https://discuss.elastic.co/t/score-is-lower-if-text-is-longer/6007/1 "2011-11-29T11:25:55Z")
Hi! I indexed a lot of documents. Two are very similar as they have the same name and the same city. The only difference is that on of these two has some fields with a lot of text in it. When I do a search over \_all fields I would expect that both results have a very similar score. But the one with the fields full of text has a significant lower score than the one with the short text. In my case the scores are ~0.2 for the one with long texts and ~0.6 for the one with the short text. So, how can I make sure, these two documents get a similar score? Robert. _My analyzers:_ {'analysis':{ 'analyzer':{ 'indexAnalyzer':{ 'type':'custom', 'tokenizer':'standard', 'filter':['lowercase','mynGram'] }, 'searchAnalyzer':{ 'type':'custom', 'tokenizer':'standard', 'filter':['standard','lowercase','mynGram'] } }, 'filter':{ 'mynGram':{ 'type':'nGram', 'min\_gram'2, 'max\_gram':50 } } }} _My mapping:_ { 'name':{ 'type':'string', 'include\_in\_all':true, 'boost':5, }, 'city':{ 'type':'string', 'include\_in\_all':true, }, 'someTextField':{ 'type':'string', 'include\_in\_all':true, }, 'someOtherTextField':{ 'type':'string', 'include\_in\_all':true, } } _My documents:_ { 'name':'Wirtschaftsinformatik', 'city':'Hamburg', 'someTextField':'', 'someOtherTextField':'' } { 'name':'Wirtschaftsinformatik', 'city':'Hamburg', 'someTextField':'A long text. Bla bla bla.', 'someOtherTextField':'Another long text. Bla bla bla.' } ---
**Author:** ![Robert](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robert/32/3055_2.png) [@Robert](https://discuss.elastic.co/u/Robert)\ **Post date:** [November 29, 2011, 11:30am UTC](https://discuss.elastic.co/t/score-is-lower-if-text-is-longer/6007/2 "2011-11-29T11:30:05Z")
I forgot: The Query performed on \_all fields is: 'Wirtschaftsinformatik AND Hamburg'. ---
**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\ **Post date:** [November 29, 2011, 2:08pm UTC](https://discuss.elastic.co/t/score-is-lower-if-text-is-longer/6007/3 "2011-11-29T14:08:32Z")
Hi, this is how Lucene scoring works. If you want the documents with the same name/city to score similarly you can use index time boosting for fields name/city. Regards, Lukas On Tue, Nov 29, 2011 at 12:30 PM, Robert [robert.katzki@googlemail.com](mailto:robert.katzki@googlemail.com)wrote: > I forgot: The Query performed on \_all fields is: 'Wirtschaftsinformatik > AND Hamburg'. ---
**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\ **Post date:** [November 29, 2011, 2:22pm UTC](https://discuss.elastic.co/t/score-is-lower-if-text-is-longer/6007/4 "2011-11-29T14:22:38Z")
Actually, you can also check > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. (see the "Multi Field" chapter). On Tue, Nov 29, 2011 at 3:08 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com) wrote: > Hi, > > this is how Lucene scoring works. If you want the documents with the same > name/city to score similarly you can use index time boosting for fields > name/city. > > Regards, > Lukas > > On Tue, Nov 29, 2011 at 12:30 PM, Robert [robert.katzki@googlemail.com](mailto:robert.katzki@googlemail.com)wrote: > > > I forgot: The Query performed on \_all fields is: 'Wirtschaftsinformatik > > AND Hamburg'. ---
**Author:** ![Robert](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robert/32/3055_2.png) [@Robert](https://discuss.elastic.co/u/Robert)\ **Post date:** [November 29, 2011, 2:39pm UTC](https://discuss.elastic.co/t/score-is-lower-if-text-is-longer/6007/5 "2011-11-29T14:39:27Z")
Hi! I already boost the fields as necessary. Name for example is way more important than the other fields. One solution that I have in mind would be to calculate a boost for each document depending on the length of the texts. But that seems not good to me... Will try out the dis\_max and tell if it helped. Any other ideas? ---
**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\ **Post date:** [November 29, 2011, 3:18pm UTC](https://discuss.elastic.co/t/score-is-lower-if-text-is-longer/6007/6 "2011-11-29T15:18:39Z")
Hey, I am not sure if boosting based on the length of the text is a good approach, I think this approach could quickly get complicated when you try to use more advanced queries. But do not take me for granted, I am not expert on Lucene Similarity [http://lucene.apache.org/java/3\_4\_0/api/all/org/apache/lucene/search/Similarity.html](http://lucene.apache.org/java/3_4_0/api/all/org/apache/lucene/search/Similarity.html) which btw should take account on length of the text, but as far as I know, by default Lucene scores shorter texts higher. I think what you need is to answer yourself a simple question: "What is the relevancy in my domain model?" Once you know the answer you can try to tackle it trying various options (boosted fields, dis max, custom scoring... Just out of curiosity, when searching on \_all field did you see any effect for boosting the Name field? I remember that I noticed some issues with it but did not had a chance to nail it down (I switched to dis max instead). Regards, Lukas On Tue, Nov 29, 2011 at 3:39 PM, Robert [robert.katzki@googlemail.com](mailto:robert.katzki@googlemail.com)wrote: > Hi! > > I already boost the fields as necessary. Name for example is way more > important than the other fields. > > One solution that I have in mind would be to calculate a boost for each > document depending on the length of the texts. But that seems not good to > me... > > Will try out the dis\_max and tell if it helped. > > Any other ideas? ---
**Author:** ![Robert](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robert/32/3055_2.png) [@Robert](https://discuss.elastic.co/u/Robert)\ **Post date:** [December 5, 2011, 3:55pm UTC](https://discuss.elastic.co/t/score-is-lower-if-text-is-longer/6007/7 "2011-12-05T15:55:28Z")
Finally could test it. But using dis\_max gives me no difference in the results. Not even the score changed. Still not sure how to analyze the data. I have to search over a lot of fields, can't limit it to one. Documents look like my example in my first post. Which is the best analyzer to be able to search for parts of a word, too? So for example I like to find the two docs from my first post when searching for 'Wirtschaft'. Documents that match exactly the search query should have a higher score. Stemming didn't give me the wanted results, that's why I use nGram atm. Regards, Robert. ---
**Author:** ![egaumer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/egaumer/32/2365_2.png) [@egaumer](https://discuss.elastic.co/u/egaumer)\ **Post date:** [December 5, 2011, 5:48pm UTC](https://discuss.elastic.co/t/score-is-lower-if-text-is-longer/6007/8 "2011-12-05T17:48:12Z")
The omit\_norms mapping attribute allows you to disable length normalization and index-time boosting for the field. This means the underlying Lucene scoring algorithm will not be effected by the length of the text associated with the given field. [http://www.elasticsearch.org/guide/reference/mapping/core-types.html](http://www.elasticsearch.org/guide/reference/mapping/core-types.html) -Eric ---
**Author:** ![Robert](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robert/32/3055_2.png) [@Robert](https://discuss.elastic.co/u/Robert)\ **Post date:** [December 7, 2011, 11:14am UTC](https://discuss.elastic.co/t/score-is-lower-if-text-is-longer/6007/9 "2011-12-07T11:14:51Z")
It doesn't give me any difference in the results. Maybe because I'm using nGram Analyzer? Any suggestions on which analyzer and which filters to use? Thanks, Robert. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:46am UTC](https://discuss.elastic.co/t/score-is-lower-if-text-is-longer/6007/10 "2017-07-06T03:46:14Z")