# Ignoring Density/Frequency **URL:** \ **Category:** Elasticsearch\ **Created:** [May 7, 2013, 2:05pm UTC](https://discuss.elastic.co/t/ignoring-density-frequency/11854 "2013-05-07T14:05:16Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Rob\_Styles](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rob_styles/32/1612_2.png) [@Rob\_Styles](https://discuss.elastic.co/u/Rob_Styles)\ **Post date:** [May 7, 2013, 2:05pm UTC](https://discuss.elastic.co/t/ignoring-density-frequency/11854/1 "2013-05-07T14:05:16Z")
Hi All, I'm trying to change search ranking and can't find info on how to do it. I suspect I'm using the wrong terminology. I am searching names within several million documents, say I search for "Tom OR Jones". Some of my docs have many names in, some only a few. The result is that a doc containing Aled Jones as the only name will score more highly than a doc that includes Tom Jones and a handful of other names as well. This is expected as word density/frequency is taken into account in the ranking right? Is there a way to configure the ranking to not take into account the length of the doc in the scoring? Such that all docs containing Tom Jones would rank higher than Aled Jones regardless of the length of the doc? What's the correct terminology to be searching for these kinds of settings? thanks rob -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Randall\_McRee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/randall_mcree/32/47177_2.png) [@Randall\_McRee](https://discuss.elastic.co/u/Randall_McRee)\ **Post date:** [May 7, 2013, 5:24pm UTC](https://discuss.elastic.co/t/ignoring-density-frequency/11854/2 "2013-05-07T17:24:09Z")
You want omit\_norms: true for the field(s) whose behaviour you want to change, I guess. Dissect the "explain" part of your query and you will see the tf-idf calculations. > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. You can google for much more information. On Tue, May 7, 2013 at 7:05 AM, Rob Styles [rob@dynamicorange.com](mailto:rob@dynamicorange.com) wrote: > Hi All, > > I'm trying to change search ranking and can't find info on how to do it. I > suspect I'm using the wrong terminology. > > I am searching names within several million documents, say I search for > "Tom OR Jones". Some of my docs have many names in, some only a few. The > result is that a doc containing Aled Jones as the only name will score more > highly than a doc that includes Tom Jones and a handful of other names as > well. > > This is expected as word density/frequency is taken into account in the > ranking right? > > Is there a way to configure the ranking to not take into account the > length of the doc in the scoring? Such that all docs containing Tom Jones > would rank higher than Aled Jones regardless of the length of the doc? > > What's the correct terminology to be searching for these kinds of settings? > > thanks > > rob > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Randall\_McRee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/randall_mcree/32/47177_2.png) [@Randall\_McRee](https://discuss.elastic.co/u/Randall_McRee)\ **Post date:** [May 7, 2013, 5:31pm UTC](https://discuss.elastic.co/t/ignoring-density-frequency/11854/3 "2013-05-07T17:31:24Z")
Looked at our implementation again--there are three things relevant to your case. 1. You may need DFS\_QUERY\_THEN\_FETCH to get accurate tf-idf calculations across shards. 2. Use omit\_norms as mentioned earlier 3. Use omit\_tf. We use both for several fields. Those are the three "levers" to investigate. Hope this helps. On Tue, May 7, 2013 at 10:24 AM, Randall McRee [randall.mcree@gmail.com](mailto:randall.mcree@gmail.com)wrote: > You want omit\_norms: true for the field(s) whose behaviour you want to > change, I guess. Dissect the "explain" part of your query and you will see > the tf-idf calculations. > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/mapping/core-types/) > > You can google for much more information. > > On Tue, May 7, 2013 at 7:05 AM, Rob Styles [rob@dynamicorange.com](mailto:rob@dynamicorange.com) wrote: > > > Hi All, > > > > I'm trying to change search ranking and can't find info on how to do it. > > I suspect I'm using the wrong terminology. > > > > I am searching names within several million documents, say I search for > > "Tom OR Jones". Some of my docs have many names in, some only a few. The > > result is that a doc containing Aled Jones as the only name will score more > > highly than a doc that includes Tom Jones and a handful of other names as > > well. > > > > This is expected as word density/frequency is taken into account in the > > ranking right? > > > > Is there a way to configure the ranking to not take into account the > > length of the doc in the scoring? Such that all docs containing Tom Jones > > would rank higher than Aled Jones regardless of the length of the doc? > > > > What's the correct terminology to be searching for these kinds of > > settings? > > > > thanks > > > > rob > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Rob\_Styles](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rob_styles/32/1612_2.png) [@Rob\_Styles](https://discuss.elastic.co/u/Rob_Styles)\ **Post date:** [May 8, 2013, 8:48am UTC](https://discuss.elastic.co/t/ignoring-density-frequency/11854/4 "2013-05-08T08:48:33Z")
Fantastic - thanks for the pointers 🙂 rob On Tuesday, May 7, 2013 6:31:24 PM UTC+1, RKM wrote: > Looked at our implementation again--there are three things relevant to > your case. > > 1. You may need DFS\_QUERY\_THEN\_FETCH to get accurate tf-idf calculations > across shards. > 2. Use omit\_norms as mentioned earlier > 3. Use omit\_tf. We use both for several fields. > > Those are the three "levers" to investigate. > > Hope this helps. > > On Tue, May 7, 2013 at 10:24 AM, Randall McRee \<[randal...@gmail.com](mailto:randal...@gmail.com)\ > > > wrote: > > > You want omit\_norms: true for the field(s) whose behaviour you want to > > change, I guess. Dissect the "explain" part of your query and you will see > > the tf-idf calculations. > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/mapping/core-types/) > > > > You can google for much more information. > > > > On Tue, May 7, 2013 at 7:05 AM, Rob Styles \<[r...@dynamicorange.com](mailto:r...@dynamicorange.com)\ > > > > > wrote: > > > > > Hi All, > > > > > > I'm trying to change search ranking and can't find info on how to do it. > > > I suspect I'm using the wrong terminology. > > > > > > I am searching names within several million documents, say I search for > > > "Tom OR Jones". Some of my docs have many names in, some only a few. The > > > result is that a doc containing Aled Jones as the only name will score more > > > highly than a doc that includes Tom Jones and a handful of other names as > > > well. > > > > > > This is expected as word density/frequency is taken into account in the > > > ranking right? > > > > > > Is there a way to configure the ranking to not take into account the > > > length of the doc in the scoring? Such that all docs containing Tom Jones > > > would rank higher than Aled Jones regardless of the length of the doc? > > > > > > What's the correct terminology to be searching for these kinds of > > > settings? > > > > > > thanks > > > > > > rob > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:37am UTC](https://discuss.elastic.co/t/ignoring-density-frequency/11854/5 "2017-07-06T02:37:41Z")
Closed on July 6, 2017, 2:37am UTC