# Controlling IDF calculation to use num docs of matched type rather than all docs in Index **URL:** \ **Category:** Elasticsearch\ **Created:** [January 6, 2014, 2:42am UTC](https://discuss.elastic.co/t/controlling-idf-calculation-to-use-num-docs-of-matched-type-rather-than-all-docs-in-index/15111 "2014-01-06T02:42:55Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Michael\_Lawler](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michael_lawler/32/128870_2.png) [@Michael\_Lawler](https://discuss.elastic.co/u/Michael_Lawler)\ **Post date:** [January 6, 2014, 2:42am UTC](https://discuss.elastic.co/t/controlling-idf-calculation-to-use-num-docs-of-matched-type-rather-than-all-docs-in-index/15111/1 "2014-01-06T02:42:55Z")
Hi, Subject line says it all. Is there an easy way to change the IDF calculation so that it uses the number of documents of the matched type (or of a predetermined type specified at query time) rather than the total number of all documents in the index. I am using documents with parent/child relationships, and as we scale up, we could have orders of magnitude more children than the parents which are important for IDF scoring. There are other reasons why we can't use separate indexes to control this. regards, Michael -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/3c6580a1-3749-4e46-ab4d-f2beaded006a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3c6580a1-3749-4e46-ab4d-f2beaded006a%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [January 6, 2014, 11:01am UTC](https://discuss.elastic.co/t/controlling-idf-calculation-to-use-num-docs-of-matched-type-rather-than-all-docs-in-index/15111/2 "2014-01-06T11:01:03Z")
Hi Michael, Unfortunately this is not possible. Maybe you could use different field names for parents and children to work around this issue? On Mon, Jan 6, 2014 at 3:42 AM, Michael Lawler [michael@lawler.id.au](mailto:michael@lawler.id.au) wrote: > Hi, > > Subject line says it all. Is there an easy way to change the IDF > calculation so that it uses the number of documents of the matched type (or > of a predetermined type specified at query time) rather than the total > number of all documents in the index. > > I am using documents with parent/child relationships, and as we scale up, > we could have orders of magnitude more children than the parents which are > important for IDF scoring. > > There are other reasons why we can't use separate indexes to control this. > > regards, > Michael > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/3c6580a1-3749-4e46-ab4d-f2beaded006a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3c6580a1-3749-4e46-ab4d-f2beaded006a%40googlegroups.com) > . > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6BwvEhdUY532Ro67BovJmU24PGy%3DV7jLkWgXR9yv-wRw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6BwvEhdUY532Ro67BovJmU24PGy%3DV7jLkWgXR9yv-wRw%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Michael\_Lawler](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michael_lawler/32/128870_2.png) [@Michael\_Lawler](https://discuss.elastic.co/u/Michael_Lawler)\ **Post date:** [January 6, 2014, 8:12pm UTC](https://discuss.elastic.co/t/controlling-idf-calculation-to-use-num-docs-of-matched-type-rather-than-all-docs-in-index/15111/3 "2014-01-06T20:12:20Z")
Hi Adrien, Thanks for your comment. Yes, I am already using different fields names across parents/child where I want to isolate IDF from each other. However my understanding of my particular issue here is that its not about the isolation provided by the field names, its about the maxDocs number that is a divisor in the maths calculating the IDF. The maxDocs number being used is the total number of documents in the index. I want it to use the total number of documents of a particular document type. Michael On Monday, January 6, 2014 10:01:03 PM UTC+11, Adrien Grand wrote: > Hi Michael, > > Unfortunately this is not possible. Maybe you could use different field > names for parents and children to work around this issue? > > On Mon, Jan 6, 2014 at 3:42 AM, Michael Lawler \ > > > wrote: > > > Hi, > > > > Subject line says it all. Is there an easy way to change the IDF > > calculation so that it uses the number of documents of the matched type (or > > of a predetermined type specified at query time) rather than the total > > number of all documents in the index. > > > > I am using documents with parent/child relationships, and as we scale up, > > we could have orders of magnitude more children than the parents which are > > important for IDF scoring. > > > > There are other reasons why we can't use separate indexes to control this. > > > > regards, > > Michael > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/3c6580a1-3749-4e46-ab4d-f2beaded006a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3c6580a1-3749-4e46-ab4d-f2beaded006a%40googlegroups.com) > > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/59d5b303-b5f5-4274-8402-2c28463ac8a2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/59d5b303-b5f5-4274-8402-2c28463ac8a2%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Riyaz](https://avatars.discourse-cdn.com/v4/letter/r/ba8739/32.png) [@Riyaz](https://discuss.elastic.co/u/Riyaz)\ **Post date:** [June 3, 2014, 9:55pm UTC](https://discuss.elastic.co/t/controlling-idf-calculation-to-use-num-docs-of-matched-type-rather-than-all-docs-in-index/15111/4 "2014-06-03T21:55:14Z")
Facing the same issue. Is there a way to make elasticsearch compute idf with maxDoc = instead of maxDoc = On Monday, January 6, 2014 3:12:20 PM UTC-5, Michael Lawler wrote: > Hi Adrien, > > Thanks for your comment. > > Yes, I am already using different fields names across parents/child where > I want to isolate IDF from each other. > > However my understanding of my particular issue here is that its not about > the isolation provided by the field names, its about the maxDocs number > that is a divisor in the maths calculating the IDF. The maxDocs number > being used is the total number of documents in the index. I want it to use > the total number of documents of a particular document type. > > Michael > > On Monday, January 6, 2014 10:01:03 PM UTC+11, Adrien Grand wrote: > > > Hi Michael, > > > > Unfortunately this is not possible. Maybe you could use different field > > names for parents and children to work around this issue? > > > > On Mon, Jan 6, 2014 at 3:42 AM, Michael Lawler [mic...@lawler.id.au](mailto:mic...@lawler.id.au) > > wrote: > > > > > Hi, > > > > > > Subject line says it all. Is there an easy way to change the IDF > > > calculation so that it uses the number of documents of the matched type (or > > > of a predetermined type specified at query time) rather than the total > > > number of all documents in the index. > > > > > > I am using documents with parent/child relationships, and as we scale > > > up, we could have orders of magnitude more children than the parents which > > > are important for IDF scoring. > > > > > > There are other reasons why we can't use separate indexes to control > > > this. > > > > > > regards, > > > Michael > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/3c6580a1-3749-4e46-ab4d-f2beaded006a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3c6580a1-3749-4e46-ab4d-f2beaded006a%40googlegroups.com) > > > . > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/331c711d-97f6-4199-b6d6-1ad7869e87db%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/331c711d-97f6-4199-b6d6-1ad7869e87db%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Riyaz](https://avatars.discourse-cdn.com/v4/letter/r/ba8739/32.png) [@Riyaz](https://discuss.elastic.co/u/Riyaz)\ **Post date:** [June 3, 2014, 9:58pm UTC](https://discuss.elastic.co/t/controlling-idf-calculation-to-use-num-docs-of-matched-type-rather-than-all-docs-in-index/15111/5 "2014-06-03T21:58:56Z")
Facing the same issue. Is there a way to make elasticsearch compute idf with maxDoc = instead of maxDoc = On Monday, January 6, 2014 3:12:20 PM UTC-5, Michael Lawler wrote: > Hi Adrien, > > Thanks for your comment. > > Yes, I am already using different fields names across parents/child where > I want to isolate IDF from each other. > > However my understanding of my particular issue here is that its not about > the isolation provided by the field names, its about the maxDocs number > that is a divisor in the maths calculating the IDF. The maxDocs number > being used is the total number of documents in the index. I want it to use > the total number of documents of a particular document type. > > Michael > > On Monday, January 6, 2014 10:01:03 PM UTC+11, Adrien Grand wrote: > > > Hi Michael, > > > > Unfortunately this is not possible. Maybe you could use different field > > names for parents and children to work around this issue? > > > > On Mon, Jan 6, 2014 at 3:42 AM, Michael Lawler [mic...@lawler.id.au](mailto:mic...@lawler.id.au) > > wrote: > > > > > Hi, > > > > > > Subject line says it all. Is there an easy way to change the IDF > > > calculation so that it uses the number of documents of the matched type (or > > > of a predetermined type specified at query time) rather than the total > > > number of all documents in the index. > > > > > > I am using documents with parent/child relationships, and as we scale > > > up, we could have orders of magnitude more children than the parents which > > > are important for IDF scoring. > > > > > > There are other reasons why we can't use separate indexes to control > > > this. > > > > > > regards, > > > Michael > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/3c6580a1-3749-4e46-ab4d-f2beaded006a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3c6580a1-3749-4e46-ab4d-f2beaded006a%40googlegroups.com) > > > . > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/94d13530-766e-4062-a3cc-7635e2238a29%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/94d13530-766e-4062-a3cc-7635e2238a29%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:24am UTC](https://discuss.elastic.co/t/controlling-idf-calculation-to-use-num-docs-of-matched-type-rather-than-all-docs-in-index/15111/6 "2017-07-06T01:24:49Z")