# Efficient similarity scoring question **URL:** \ **Category:** Elasticsearch\ **Created:** [October 1, 2013, 10:46pm UTC](https://discuss.elastic.co/t/efficient-similarity-scoring-question/13821 "2013-10-01T22:46:03Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Chris\_6](https://avatars.discourse-cdn.com/v4/letter/c/e47774/32.png) [@Chris\_6](https://discuss.elastic.co/u/Chris_6)\ **Post date:** [October 1, 2013, 10:46pm UTC](https://discuss.elastic.co/t/efficient-similarity-scoring-question/13821/1 "2013-10-01T22:46:03Z")
Hi, Efficiency question for you guys. I have a set of documents with a repeated integer: { "document": { "properties": { "id": { "index": "no", "type": "string" }, "fprint": { "postings\_format": "bloom\_pulsing", "type": "long" }, "fprint\_size": { "include\_in\_all": false, "store": true, "type": "integer" }, } } } My query is a set of fingerprints, and I would like the final score to be # of matching fingerprints from the document normalized by the number of fingerprints in the document. This query retrieves the right set, but does not normalize: { "query": { "bool": { "should" : [ ``` { "term" : { "morgan_fprint" : 632180975 } }, { "term" : { "morgan_fprint" : 1039876598 } }, { "term" : { "morgan_fprint" : 2246728737 } }, { "term" : { "morgan_fprint" : 2264700157 } } ], "minimum_should_match" : 1, "boost" : 1.0 ``` } } }' The advantage of this query is that it takes only ~50ms. This query does the normalization, but is significantly slower (e.g. 250ms): { "query": { "custom\_score": { "script" : "(\_score / doc.morgan\_fprint\_size.value)", "query": { ``` "bool": { "should" : [ { "term" : { "morgan_fprint" : 632180975 } }, { "term" : { "morgan_fprint" : 1039876598 } }, { "term" : { "morgan_fprint" : 2246728737 } }, { "term" : { "morgan_fprint" : 2264700157 } } ], "minimum_should_match" : 1, "boost" : 1.0 } } ``` } } } ==== I assume this is because of how morgan\_fprint\_size.value is retrieved from disk during query execution. Is there a good way of structuring the index such that I can get both a fast, normalized query? I attempted to do an index side, per-document boost (e.g. something like "fprint": { "\_value": 12345, "\_value": 67890, "\_boost": 0.5 }). However I got this error: "You cannot set an index-time boost on an unindexed field, or one that omits norms" ... So that didn't work. Thanks, Chris -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Chris\_6](https://avatars.discourse-cdn.com/v4/letter/c/e47774/32.png) [@Chris\_6](https://discuss.elastic.co/u/Chris_6)\ **Post date:** [October 1, 2013, 10:47pm UTC](https://discuss.elastic.co/t/efficient-similarity-scoring-question/13821/2 "2013-10-01T22:47:28Z")
And sorry for the confusion with "fprint" vs "morgan\_fprint". I had a copy pasting error, please ignore the "morgan\_" prefix. - Chris On Tuesday, October 1, 2013 3:46:03 PM UTC-7, Chris wrote: > Hi, > > Efficiency question for you guys. > > I have a set of documents with a repeated integer: > { > "document": { > "properties": { > "id": { > "index": "no", > "type": "string" > }, > "fprint": { > "postings\_format": "bloom\_pulsing", > "type": "long" > }, > "fprint\_size": { > "include\_in\_all": false, > "store": true, > "type": "integer" > }, > } > } > } > > My query is a set of fingerprints, and I would like the final score to be > > # of matching fingerprints from the document normalized by the number of > > fingerprints in the document. This query retrieves the right set, but does > not normalize: > > { "query": { > > "bool": { > > "should" : [ > > ``` > { > > "term" : { "morgan_fprint" : 632180975 } > > }, > > { > > "term" : { "morgan_fprint" : 1039876598 } > > }, > > { > > "term" : { "morgan_fprint" : 2246728737 } > > }, > > { > > "term" : { "morgan_fprint" : 2264700157 } > > } > > ], > > "minimum_should_match" : 1, > > "boost" : 1.0 > > ``` > > } > > } > }' > > The advantage of this query is that it takes only ~50ms. > > This query does the normalization, but is significantly slower (e.g. > 250ms): > > { "query": { > > "custom\_score": { > > "script" : "(\_score / doc.morgan\_fprint\_size.value)", > > "query": { > > ``` > "bool": { > > "should" : [ > > { > > "term" : { "morgan_fprint" : 632180975 } > > }, > > { > > "term" : { "morgan_fprint" : 1039876598 } > > }, > > { > > "term" : { "morgan_fprint" : 2246728737 } > > }, > > { > > "term" : { "morgan_fprint" : 2264700157 } > > } > > ], > > "minimum_should_match" : 1, > > "boost" : 1.0 > > } > > } > > ``` > > } > > } } > > ==== > > I assume this is because of how morgan\_fprint\_size.value is retrieved from > disk during query execution. Is there a good way of structuring the index > such that I can get both a fast, normalized query? > > I attempted to do an index side, per-document boost (e.g. something like > "fprint": { "\_value": 12345, "\_value": 67890, "\_boost": 0.5 }). However I > got this error: > "You cannot set an index-time boost on an unindexed field, or one that > omits norms" > > ... So that didn't work. > > Thanks, > Chris -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![benjamin\_leviant](https://avatars.discourse-cdn.com/v4/letter/b/919ad9/32.png) [@benjamin\_leviant](https://discuss.elastic.co/u/benjamin_leviant)\ **Post date:** [October 4, 2013, 8:19am UTC](https://discuss.elastic.co/t/efficient-similarity-scoring-question/13821/3 "2013-10-04T08:19:42Z")
Hi Chris, The numerical fields are not normalized. If you want to used the default normalization provided by the lucene index (that should be much faster), you should index your fprint values as string. If in your case the numerical format is mandatory you can use multi\_field to index your field both as string and as long types. Be warned that the default normalization is not very precise. The small field length differences may not be taken in account in the final scoring. To avoid this problem, I can see 2 solutions : - You can try to amplify the norm variation by using field boost like you did previously. But before you will need to understand how lucene process field boost. For more info on this subject : [https://lucene.apache.org/core/3\_6\_2/api/core/org/apache/lucene/document/Fieldable.html#setBoost(float)](https://lucene.apache.org/core/3_6_2/api/core/org/apache/lucene/document/Fieldable.html#setBoost(float)) The correct syntax will be something like this : { "fprint.stringify": [ { "\_value": "12345", "\_boost": 0.5 }, { "\_value": "67890", "\_boost": 0.5 } ] } - An other solution should be to use a custom similarity class that use a more precise normalization, but this not trivial and will required to understand how lucene indexes norm fields. I hope this can help you. Regards Benjamin On Wed, Oct 2, 2013 at 12:47 AM, Chris [chris.vana@gmail.com](mailto:chris.vana@gmail.com) wrote: > And sorry for the confusion with "fprint" vs "morgan\_fprint". I had a copy > pasting error, please ignore the "morgan\_" prefix. > > - Chris > > On Tuesday, October 1, 2013 3:46:03 PM UTC-7, Chris wrote: > > > Hi, > > > > Efficiency question for you guys. > > > > I have a set of documents with a repeated integer: > > { > > "document": { > > "properties": { > > "id": { > > "index": "no", > > "type": "string" > > }, > > "fprint": { > > "postings\_format": "bloom\_pulsing", > > "type": "long" > > }, > > "fprint\_size": { > > "include\_in\_all": false, > > "store": true, > > "type": "integer" > > }, > > } > > } > > } > > > > My query is a set of fingerprints, and I would like the final score to be > > > > # of matching fingerprints from the document normalized by the number of > > > > fingerprints in the document. This query retrieves the right set, but does > > not normalize: > > > > { "query": { > > > > ``` > > "bool": { > > > > ``` > > > > "should" : [ > > > > ``` > > { > > > > "term" : { "morgan_fprint" : 632180975 } > > > > }, > > > > { > > > > "term" : { "morgan_fprint" : 1039876598 } > > > > }, > > > > { > > > > "term" : { "morgan_fprint" : 2246728737 } > > > > }, > > > > { > > > > "term" : { "morgan_fprint" : 2264700157 } > > > > } > > > > ], > > > > "minimum_should_match" : 1, > > > > "boost" : 1.0 > > > > } > > > > ``` > > > > } > > }' > > > > The advantage of this query is that it takes only ~50ms. > > > > This query does the normalization, but is significantly slower (e.g. > > 250ms): > > > > { "query": { > > > > "custom\_score": { > > > > ``` > > "script" : "(_score / doc.morgan_fprint_size.value)"**, > > > > "query": { > > > > "bool": { > > > > "should" : [ > > > > { > > > > "term" : { "morgan_fprint" : 632180975 } > > > > }, > > > > { > > > > "term" : { "morgan_fprint" : 1039876598 } > > > > }, > > > > { > > > > "term" : { "morgan_fprint" : 2246728737 } > > > > }, > > > > { > > > > "term" : { "morgan_fprint" : 2264700157 } > > > > } > > > > ], > > > > "minimum_should_match" : 1, > > > > "boost" : 1.0 > > > > } > > > > } > > > > ``` > > > > } > > > > } } > > > > ==== > > > > I assume this is because of how morgan\_fprint\_size.value is retrieved > > from disk during query execution. Is there a good way of structuring the > > index such that I can get both a fast, normalized query? > > > > I attempted to do an index side, per-document boost (e.g. something like > > "fprint": { "\_value": 12345, "\_value": 67890, "\_boost": 0.5 }). However I > > got this error: > > "You cannot set an index-time boost on an unindexed field, or one that > > omits norms" > > > > ... So that didn't work. > > > > Thanks, > > Chris > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:13am UTC](https://discuss.elastic.co/t/efficient-similarity-scoring-question/13821/4 "2017-07-06T02:13:49Z")