# Modification of term weight **URL:** \ **Category:** Elasticsearch\ **Created:** [September 9, 2010, 10:57pm UTC](https://discuss.elastic.co/t/modification-of-term-weight/3318 "2010-09-09T22:57:15Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Michael\_Korbakov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michael_korbakov/32/3199_2.png) [@Michael\_Korbakov](https://discuss.elastic.co/u/Michael_Korbakov)\ **Post date:** [September 9, 2010, 10:57pm UTC](https://discuss.elastic.co/t/modification-of-term-weight/3318/1 "2010-09-09T22:57:15Z")
Hi everybody. Is there any way to change term weight in the context of one particular document? Or, other words, to change TF value of specific term for specific document? I need this trick to promote specific documents in the context of predefined queries. May be there's some better way to achieve it? I just can think any other then TF manipulation. Thank you in advance, -- Michael Korbakov ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [September 10, 2010, 1:21pm UTC](https://discuss.elastic.co/t/modification-of-term-weight/3318/2 "2010-09-10T13:21:32Z")
It really depends on what exactly you are after, but custom\_score query might help here: [http://www.elasticsearch.com/docs/elasticsearch/rest\_api/query\_dsl/custom\_score\_query/](http://www.elasticsearch.com/docs/elasticsearch/rest_api/query_dsl/custom_score_query/) . On Fri, Sep 10, 2010 at 1:57 AM, rmihael [rmihael@gmail.com](mailto:rmihael@gmail.com) wrote: > Hi everybody. > > Is there any way to change term weight in the context of one > particular document? Or, other words, to change TF value of specific > term for specific document? > I need this trick to promote specific documents in the context of > predefined queries. May be there's some better way to achieve it? I > just can think any other then TF manipulation. > > Thank you in advance, > > -- Michael Korbakov ---
**Author:** ![Michael\_Korbakov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michael_korbakov/32/3199_2.png) [@Michael\_Korbakov](https://discuss.elastic.co/u/Michael_Korbakov)\ **Post date:** [September 10, 2010, 1:39pm UTC](https://discuss.elastic.co/t/modification-of-term-weight/3318/3 "2010-09-10T13:39:45Z")
Unfortunately it's not my case. My goal is to make changes like this: "set weight of term 'woman' in document with id 1231231231 to 1.5". I don't think that it's possible with custom\_score, or at least I'm missing something important about it. On Fri, Sep 10, 2010 at 4:21 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > It really depends on what exactly you are after, but custom\_score query > might help here: > [http://www.elasticsearch.com/docs/elasticsearch/rest\_api/query\_dsl/custom\_score\_query/](http://www.elasticsearch.com/docs/elasticsearch/rest_api/query_dsl/custom_score_query/) > . > > On Fri, Sep 10, 2010 at 1:57 AM, rmihael [rmihael@gmail.com](mailto:rmihael@gmail.com) wrote: > > > Hi everybody. > > > > Is there any way to change term weight in the context of one > > particular document? Or, other words, to change TF value of specific > > term for specific document? > > I need this trick to promote specific documents in the context of > > predefined queries. May be there's some better way to achieve it? I > > just can think any other then TF manipulation. > > > > Thank you in advance, > > > > -- Michael Korbakov ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [September 10, 2010, 1:42pm UTC](https://discuss.elastic.co/t/modification-of-term-weight/3318/4 "2010-09-10T13:42:22Z")
No, you can't do that. But you can control it on the search aspect, by constructing a boosted query and filtered by specific docs. This solution really depends on the number of custom weights you wish to set though... . On Fri, Sep 10, 2010 at 4:39 PM, Mykhailo Korbakov [rmihael@gmail.com](mailto:rmihael@gmail.com)wrote: > Unfortunately it's not my case. > > My goal is to make changes like this: "set weight of term 'woman' in > document with id 1231231231 to 1.5". I don't think that it's possible with > custom\_score, or at least I'm missing something important about it. > > On Fri, Sep 10, 2010 at 4:21 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > > > It really depends on what exactly you are after, but custom\_score query > > might help here: > > [http://www.elasticsearch.com/docs/elasticsearch/rest\_api/query\_dsl/custom\_score\_query/](http://www.elasticsearch.com/docs/elasticsearch/rest_api/query_dsl/custom_score_query/) > > . > > > > On Fri, Sep 10, 2010 at 1:57 AM, rmihael [rmihael@gmail.com](mailto:rmihael@gmail.com) wrote: > > > > > Hi everybody. > > > > > > Is there any way to change term weight in the context of one > > > particular document? Or, other words, to change TF value of specific > > > term for specific document? > > > I need this trick to promote specific documents in the context of > > > predefined queries. May be there's some better way to achieve it? I > > > just can think any other then TF manipulation. > > > > > > Thank you in advance, > > > > > > -- Michael Korbakov ---
**Author:** ![Michael\_Korbakov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michael_korbakov/32/3199_2.png) [@Michael\_Korbakov](https://discuss.elastic.co/u/Michael_Korbakov)\ **Post date:** [September 10, 2010, 2:00pm UTC](https://discuss.elastic.co/t/modification-of-term-weight/3318/5 "2010-09-10T14:00:04Z")
Oh I wish that this weird trick would be _my_ wish 🙂 I can come up with solution for few of these corrected weights, but unfortunately there's a plan for thousands of them. Don't you think that term weight access (both TF and IDF parts) can be useful in ES? Having some way to modify IDF weights is really useful for advanced ranking algorithms with implicit feedback processing. As for now we have to pass terms weights specified in queries and it doesn't very convenient. May be it is worth to open an issue in GitHub? On Fri, Sep 10, 2010 at 4:42 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > No, you can't do that. But you can control it on the search aspect, by > constructing a boosted query and filtered by specific docs. This solution > really depends on the number of custom weights you wish to set though... . > > On Fri, Sep 10, 2010 at 4:39 PM, Mykhailo Korbakov [rmihael@gmail.com](mailto:rmihael@gmail.com)wrote: > > > Unfortunately it's not my case. > > > > My goal is to make changes like this: "set weight of term 'woman' in > > document with id 1231231231 to 1.5". I don't think that it's possible with > > custom\_score, or at least I'm missing something important about it. > > > > On Fri, Sep 10, 2010 at 4:21 PM, Shay Banon \<[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) > > > > > wrote: > > > > > It really depends on what exactly you are after, but custom\_score query > > > might help here: > > > [http://www.elasticsearch.com/docs/elasticsearch/rest\_api/query\_dsl/custom\_score\_query/](http://www.elasticsearch.com/docs/elasticsearch/rest_api/query_dsl/custom_score_query/) > > > . > > > > > > On Fri, Sep 10, 2010 at 1:57 AM, rmihael [rmihael@gmail.com](mailto:rmihael@gmail.com) wrote: > > > > > > > Hi everybody. > > > > > > > > Is there any way to change term weight in the context of one > > > > particular document? Or, other words, to change TF value of specific > > > > term for specific document? > > > > I need this trick to promote specific documents in the context of > > > > predefined queries. May be there's some better way to achieve it? I > > > > just can think any other then TF manipulation. > > > > > > > > Thank you in advance, > > > > > > > > -- Michael Korbakov ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [September 10, 2010, 2:12pm UTC](https://discuss.elastic.co/t/modification-of-term-weight/3318/6 "2010-09-10T14:12:55Z")
Yea, I agree. Though its tricky to expose this functionality nicely... . Basically, on the indexing side, there is a possibility to control the weighting of a term for example by adding a payload to it that represent the score. And then, when searching, using a custom query that takes those payloads and adds them to the scoring. This means that when indexing, you will need to provide the weighting (or just the special ones), for example: "the women^2 went to the beach". This does mean that for each document you want to change the term weights, it will need to be indexed with a new value(s), for example: "the women^1.5 went to the beach^2". That format (^) denoting custom weight is something I invented here for the example. Then, when searching, those weights will be taken into account using a special custom\_weight query that will be created for it. Does that sound like something that will help you? On Fri, Sep 10, 2010 at 5:00 PM, Mykhailo Korbakov [rmihael@gmail.com](mailto:rmihael@gmail.com)wrote: > Oh I wish that this weird trick would be _my_ wish 🙂 > I can come up with solution for few of these corrected weights, but > unfortunately there's a plan for thousands of them. > > Don't you think that term weight access (both TF and IDF parts) can be > useful in ES? Having some way to modify IDF weights is really useful for > advanced ranking algorithms with implicit feedback processing. As for now we > have to pass terms weights specified in queries and it doesn't very > convenient. > > May be it is worth to open an issue in GitHub? > > On Fri, Sep 10, 2010 at 4:42 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > > > No, you can't do that. But you can control it on the search aspect, by > > constructing a boosted query and filtered by specific docs. This solution > > really depends on the number of custom weights you wish to set though... . > > > > On Fri, Sep 10, 2010 at 4:39 PM, Mykhailo Korbakov [rmihael@gmail.com](mailto:rmihael@gmail.com)wrote: > > > > > Unfortunately it's not my case. > > > > > > My goal is to make changes like this: "set weight of term 'woman' in > > > document with id 1231231231 to 1.5". I don't think that it's possible with > > > custom\_score, or at least I'm missing something important about it. > > > > > > On Fri, Sep 10, 2010 at 4:21 PM, Shay Banon \< > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote: > > > > > > > It really depends on what exactly you are after, but custom\_score query > > > > might help here: > > > > [http://www.elasticsearch.com/docs/elasticsearch/rest\_api/query\_dsl/custom\_score\_query/](http://www.elasticsearch.com/docs/elasticsearch/rest_api/query_dsl/custom_score_query/) > > > > . > > > > > > > > On Fri, Sep 10, 2010 at 1:57 AM, rmihael [rmihael@gmail.com](mailto:rmihael@gmail.com) wrote: > > > > > > > > > Hi everybody. > > > > > > > > > > Is there any way to change term weight in the context of one > > > > > particular document? Or, other words, to change TF value of specific > > > > > term for specific document? > > > > > I need this trick to promote specific documents in the context of > > > > > predefined queries. May be there's some better way to achieve it? I > > > > > just can think any other then TF manipulation. > > > > > > > > > > Thank you in advance, > > > > > > > > > > -- Michael Korbakov ---
**Author:** ![Michael\_Korbakov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michael_korbakov/32/3199_2.png) [@Michael\_Korbakov](https://discuss.elastic.co/u/Michael_Korbakov)\ **Post date:** [September 21, 2010, 10:56am UTC](https://discuss.elastic.co/t/modification-of-term-weight/3318/7 "2010-09-21T10:56:04Z")
Hi Shay. Sorry for long delay, I was out of town. I believe that this problem splits into two: 1. Controlling index-wide terms weight, let's call it IDF. This is the feature with immediate influence on practical applications as it makes introduction of ranking analysis algorithms like SVMRank much easier and strain forward. I don't think however that it should be kept in indexing step. That solution simply breaks the logic: IDF is index-wide and indexing document is "document-wide". I believe that ES needs some terms API that can be used to query for indexed terms information and also for setting their personal weights. One particular problem I can foresee here is text analysis. It's hard to predict how exactly terms from my documents will be transformed during indexing. May be it's worth to return a set of analyzed terms as response for document indexing? 2. Controlling document-wide weights of terms, let's call it TF. Although our conversation started with it, I believe that it much less important and useful then index-wide IDF. Your solution with "the women^1.5 went to the beach^2" is very nice, but I have one concern: how can it play with contradictive inputs? For instance "cat^1 cats^2" in analyzed field. "cat" and "cats" will be analyzed into one single term that will have two weights (1, 2). Out of ideas how to design it in proper way. BTW, is there any way to set all IDF weights for some particular field to 1.0? I found that "omit\_term\_freq\_and\_positions" parameter disables TF, but how to turn off IDF? On Fri, Sep 10, 2010 at 5:12 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > Yea, I agree. Though its tricky to expose this functionality nicely... . > Basically, on the indexing side, there is a possibility to control the > weighting of a term for example by adding a payload to it that represent the > score. And then, when searching, using a custom query that takes those > payloads and adds them to the scoring. This means that when indexing, you > will need to provide the weighting (or just the special ones), for example: > "the women^2 went to the beach". This does mean that for each document you > want to change the term weights, it will need to be indexed with a new > value(s), for example: "the women^1.5 went to the beach^2". That format (^) > denoting custom weight is something I invented here for the example. Then, > when searching, those weights will be taken into account using a special > custom\_weight query that will be created for it. Does that sound like > something that will help you? > > On Fri, Sep 10, 2010 at 5:00 PM, Mykhailo Korbakov [rmihael@gmail.com](mailto:rmihael@gmail.com)wrote: > > > Oh I wish that this weird trick would be _my_ wish 🙂 > > I can come up with solution for few of these corrected weights, but > > unfortunately there's a plan for thousands of them. > > > > Don't you think that term weight access (both TF and IDF parts) can be > > useful in ES? Having some way to modify IDF weights is really useful for > > advanced ranking algorithms with implicit feedback processing. As for now we > > have to pass terms weights specified in queries and it doesn't very > > convenient. > > > > May be it is worth to open an issue in GitHub? > > > > On Fri, Sep 10, 2010 at 4:42 PM, Shay Banon \<[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) > > > > > wrote: > > > > > No, you can't do that. But you can control it on the search aspect, by > > > constructing a boosted query and filtered by specific docs. This solution > > > really depends on the number of custom weights you wish to set though... . > > > > > > On Fri, Sep 10, 2010 at 4:39 PM, Mykhailo Korbakov [rmihael@gmail.com](mailto:rmihael@gmail.com)wrote: > > > > > > > Unfortunately it's not my case. > > > > > > > > My goal is to make changes like this: "set weight of term 'woman' in > > > > document with id 1231231231 to 1.5". I don't think that it's possible with > > > > custom\_score, or at least I'm missing something important about it. > > > > > > > > On Fri, Sep 10, 2010 at 4:21 PM, Shay Banon \< > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote: > > > > > > > > > It really depends on what exactly you are after, but custom\_score query > > > > > might help here: > > > > > [http://www.elasticsearch.com/docs/elasticsearch/rest\_api/query\_dsl/custom\_score\_query/](http://www.elasticsearch.com/docs/elasticsearch/rest_api/query_dsl/custom_score_query/) > > > > > . > > > > > > > > > > On Fri, Sep 10, 2010 at 1:57 AM, rmihael [rmihael@gmail.com](mailto:rmihael@gmail.com) wrote: > > > > > > > > > > > Hi everybody. > > > > > > > > > > > > Is there any way to change term weight in the context of one > > > > > > particular document? Or, other words, to change TF value of specific > > > > > > term for specific document? > > > > > > I need this trick to promote specific documents in the context of > > > > > > predefined queries. May be there's some better way to achieve it? I > > > > > > just can think any other then TF manipulation. > > > > > > > > > > > > Thank you in advance, > > > > > > > > > > > > -- Michael Korbakov ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [September 23, 2010, 4:49pm UTC](https://discuss.elastic.co/t/modification-of-term-weight/3318/8 "2010-09-23T16:49:20Z")
Hey, What you are asking for is quite advanced, and there are things that you can do and things that you simply can't mainly because of how Lucene, the underlying search library elasticsearch uses is built. For example, being able to change in runtime the weight of a term is not something that is possible in Lucene. I suggest you read this page which gives a good overview of how Lucene does its scoring: [Apache Lucene - Scoring](http://lucene.apache.org/java/3_0_2/scoring.html), and we can continue the discussion from there? -shay.banon On Tue, Sep 21, 2010 at 12:56 PM, Mykhailo Korbakov [rmihael@gmail.com](mailto:rmihael@gmail.com)wrote: > Hi Shay. > > Sorry for long delay, I was out of town. I believe that this problem splits > into two: > > 1. Controlling index-wide terms weight, let's call it IDF. This is the > feature with immediate influence on practical applications as it makes > introduction of ranking analysis algorithms like SVMRank much easier and > strain forward. I don't think however that it should be kept in indexing > step. That solution simply breaks the logic: IDF is index-wide and indexing > document is "document-wide". I believe that ES needs some terms API that can > be used to query for indexed terms information and also for setting their > personal weights. > One particular problem I can foresee here is text analysis. It's hard to > predict how exactly terms from my documents will be transformed during > indexing. May be it's worth to return a set of analyzed terms as response > for document indexing? > > 2. Controlling document-wide weights of terms, let's call it TF. Although > our conversation started with it, I believe that it much less important and > useful then index-wide IDF. Your solution with "the women^1.5 went to the > beach^2" is very nice, but I have one concern: how can it play with > contradictive inputs? For instance "cat^1 cats^2" in analyzed field. "cat" > and "cats" will be analyzed into one single term that will have two weights > (1, 2). Out of ideas how to design it in proper way. > > BTW, is there any way to set all IDF weights for some particular field to > 1.0? I found that "omit\_term\_freq\_and\_positions" parameter disables TF, but > how to turn off IDF? > > On Fri, Sep 10, 2010 at 5:12 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > > > Yea, I agree. Though its tricky to expose this functionality nicely... . > > Basically, on the indexing side, there is a possibility to control the > > weighting of a term for example by adding a payload to it that represent the > > score. And then, when searching, using a custom query that takes those > > payloads and adds them to the scoring. This means that when indexing, you > > will need to provide the weighting (or just the special ones), for example: > > "the women^2 went to the beach". This does mean that for each document you > > want to change the term weights, it will need to be indexed with a new > > value(s), for example: "the women^1.5 went to the beach^2". That format (^) > > denoting custom weight is something I invented here for the example. Then, > > when searching, those weights will be taken into account using a special > > custom\_weight query that will be created for it. Does that sound like > > something that will help you? > > > > On Fri, Sep 10, 2010 at 5:00 PM, Mykhailo Korbakov [rmihael@gmail.com](mailto:rmihael@gmail.com)wrote: > > > > > Oh I wish that this weird trick would be _my_ wish 🙂 > > > I can come up with solution for few of these corrected weights, but > > > unfortunately there's a plan for thousands of them. > > > > > > Don't you think that term weight access (both TF and IDF parts) can be > > > useful in ES? Having some way to modify IDF weights is really useful for > > > advanced ranking algorithms with implicit feedback processing. As for now we > > > have to pass terms weights specified in queries and it doesn't very > > > convenient. > > > > > > May be it is worth to open an issue in GitHub? > > > > > > On Fri, Sep 10, 2010 at 4:42 PM, Shay Banon \< > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote: > > > > > > > No, you can't do that. But you can control it on the search aspect, by > > > > constructing a boosted query and filtered by specific docs. This solution > > > > really depends on the number of custom weights you wish to set though... . > > > > > > > > On Fri, Sep 10, 2010 at 4:39 PM, Mykhailo Korbakov [rmihael@gmail.com](mailto:rmihael@gmail.com)wrote: > > > > > > > > > Unfortunately it's not my case. > > > > > > > > > > My goal is to make changes like this: "set weight of term 'woman' in > > > > > document with id 1231231231 to 1.5". I don't think that it's possible with > > > > > custom\_score, or at least I'm missing something important about it. > > > > > > > > > > On Fri, Sep 10, 2010 at 4:21 PM, Shay Banon \< > > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote: > > > > > > > > > > > It really depends on what exactly you are after, but custom\_score > > > > > > query might help here: > > > > > > [http://www.elasticsearch.com/docs/elasticsearch/rest\_api/query\_dsl/custom\_score\_query/](http://www.elasticsearch.com/docs/elasticsearch/rest_api/query_dsl/custom_score_query/) > > > > > > . > > > > > > > > > > > > On Fri, Sep 10, 2010 at 1:57 AM, rmihael [rmihael@gmail.com](mailto:rmihael@gmail.com) wrote: > > > > > > > > > > > > > Hi everybody. > > > > > > > > > > > > > > Is there any way to change term weight in the context of one > > > > > > > particular document? Or, other words, to change TF value of specific > > > > > > > term for specific document? > > > > > > > I need this trick to promote specific documents in the context of > > > > > > > predefined queries. May be there's some better way to achieve it? I > > > > > > > just can think any other then TF manipulation. > > > > > > > > > > > > > > Thank you in advance, > > > > > > > > > > > > > > -- Michael Korbakov ---
**Author:** ![Michael\_Korbakov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michael_korbakov/32/3199_2.png) [@Michael\_Korbakov](https://discuss.elastic.co/u/Michael_Korbakov)\ **Post date:** [September 23, 2010, 8:27pm UTC](https://discuss.elastic.co/t/modification-of-term-weight/3318/9 "2010-09-23T20:27:35Z")
Sure, thanks for direction. Will read it now. On Thu, Sep 23, 2010 at 7:49 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > Hey, > > What you are asking for is quite advanced, and there are things that you > can do and things that you simply can't mainly because of how Lucene, the > underlying search library elasticsearch uses is built. For example, being > able to change in runtime the weight of a term is not something that is > possible in Lucene. I suggest you read this page which gives a good overview > of how Lucene does its scoring: > [Apache Lucene - Scoring](http://lucene.apache.org/java/3_0_2/scoring.html), and we can continue the > discussion from there? > > -shay.banon > > On Tue, Sep 21, 2010 at 12:56 PM, Mykhailo Korbakov [rmihael@gmail.com](mailto:rmihael@gmail.com)wrote: > > > Hi Shay. > > > > Sorry for long delay, I was out of town. I believe that this problem > > splits into two: > > > > 1. Controlling index-wide terms weight, let's call it IDF. This is the > > feature with immediate influence on practical applications as it makes > > introduction of ranking analysis algorithms like SVMRank much easier and > > strain forward. I don't think however that it should be kept in indexing > > step. That solution simply breaks the logic: IDF is index-wide and indexing > > document is "document-wide". I believe that ES needs some terms API that can > > be used to query for indexed terms information and also for setting their > > personal weights. > > One particular problem I can foresee here is text analysis. It's hard to > > predict how exactly terms from my documents will be transformed during > > indexing. May be it's worth to return a set of analyzed terms as response > > for document indexing? > > > > 2. Controlling document-wide weights of terms, let's call it TF. Although > > our conversation started with it, I believe that it much less important and > > useful then index-wide IDF. Your solution with "the women^1.5 went to the > > beach^2" is very nice, but I have one concern: how can it play with > > contradictive inputs? For instance "cat^1 cats^2" in analyzed field. "cat" > > and "cats" will be analyzed into one single term that will have two weights > > (1, 2). Out of ideas how to design it in proper way. > > > > BTW, is there any way to set all IDF weights for some particular field to > > 1.0? I found that "omit\_term\_freq\_and\_positions" parameter disables TF, but > > how to turn off IDF? > > > > On Fri, Sep 10, 2010 at 5:12 PM, Shay Banon \<[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) > > > > > wrote: > > > > > Yea, I agree. Though its tricky to expose this functionality nicely... . > > > Basically, on the indexing side, there is a possibility to control the > > > weighting of a term for example by adding a payload to it that represent the > > > score. And then, when searching, using a custom query that takes those > > > payloads and adds them to the scoring. This means that when indexing, you > > > will need to provide the weighting (or just the special ones), for example: > > > "the women^2 went to the beach". This does mean that for each document you > > > want to change the term weights, it will need to be indexed with a new > > > value(s), for example: "the women^1.5 went to the beach^2". That format (^) > > > denoting custom weight is something I invented here for the example. Then, > > > when searching, those weights will be taken into account using a special > > > custom\_weight query that will be created for it. Does that sound like > > > something that will help you? > > > > > > On Fri, Sep 10, 2010 at 5:00 PM, Mykhailo Korbakov [rmihael@gmail.com](mailto:rmihael@gmail.com)wrote: > > > > > > > Oh I wish that this weird trick would be _my_ wish 🙂 > > > > I can come up with solution for few of these corrected weights, but > > > > unfortunately there's a plan for thousands of them. > > > > > > > > Don't you think that term weight access (both TF and IDF parts) can be > > > > useful in ES? Having some way to modify IDF weights is really useful for > > > > advanced ranking algorithms with implicit feedback processing. As for now we > > > > have to pass terms weights specified in queries and it doesn't very > > > > convenient. > > > > > > > > May be it is worth to open an issue in GitHub? > > > > > > > > On Fri, Sep 10, 2010 at 4:42 PM, Shay Banon \< > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote: > > > > > > > > > No, you can't do that. But you can control it on the search aspect, by > > > > > constructing a boosted query and filtered by specific docs. This solution > > > > > really depends on the number of custom weights you wish to set though... . > > > > > > > > > > On Fri, Sep 10, 2010 at 4:39 PM, Mykhailo Korbakov [rmihael@gmail.com](mailto:rmihael@gmail.com)wrote: > > > > > > > > > > > Unfortunately it's not my case. > > > > > > > > > > > > My goal is to make changes like this: "set weight of term 'woman' in > > > > > > document with id 1231231231 to 1.5". I don't think that it's possible with > > > > > > custom\_score, or at least I'm missing something important about it. > > > > > > > > > > > > On Fri, Sep 10, 2010 at 4:21 PM, Shay Banon \< > > > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote: > > > > > > > > > > > > > It really depends on what exactly you are after, but custom\_score > > > > > > > query might help here: > > > > > > > [http://www.elasticsearch.com/docs/elasticsearch/rest\_api/query\_dsl/custom\_score\_query/](http://www.elasticsearch.com/docs/elasticsearch/rest_api/query_dsl/custom_score_query/) > > > > > > > . > > > > > > > > > > > > > > On Fri, Sep 10, 2010 at 1:57 AM, rmihael [rmihael@gmail.com](mailto:rmihael@gmail.com) wrote: > > > > > > > > > > > > > > > Hi everybody. > > > > > > > > > > > > > > > > Is there any way to change term weight in the context of one > > > > > > > > particular document? Or, other words, to change TF value of specific > > > > > > > > term for specific document? > > > > > > > > I need this trick to promote specific documents in the context of > > > > > > > > predefined queries. May be there's some better way to achieve it? I > > > > > > > > just can think any other then TF manipulation. > > > > > > > > > > > > > > > > Thank you in advance, > > > > > > > > > > > > > > > > -- Michael Korbakov ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [September 23, 2010, 9:09pm UTC](https://discuss.elastic.co/t/modification-of-term-weight/3318/10 "2010-09-23T21:09:20Z")
Great, note that all extension points (custom Similarity and so on) can be hooked into elasticsearch easily. Just ping if you want more info. As a side note, I know that there has been a lot of discussion going on in Lucene to try and open this aspect a bit more to allow for more custom implementations. On Thu, Sep 23, 2010 at 10:27 PM, Mykhailo Korbakov [rmihael@gmail.com](mailto:rmihael@gmail.com)wrote: > Sure, thanks for direction. Will read it now. > > On Thu, Sep 23, 2010 at 7:49 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > > > Hey, > > > > What you are asking for is quite advanced, and there are things that > > you can do and things that you simply can't mainly because of how Lucene, > > the underlying search library elasticsearch uses is built. For example, > > being able to change in runtime the weight of a term is not something that > > is possible in Lucene. I suggest you read this page which gives a good > > overview of how Lucene does its scoring: > > [Apache Lucene - Scoring](http://lucene.apache.org/java/3_0_2/scoring.html), and we can continue the > > discussion from there? > > > > -shay.banon > > > > On Tue, Sep 21, 2010 at 12:56 PM, Mykhailo Korbakov [rmihael@gmail.com](mailto:rmihael@gmail.com)wrote: > > > > > Hi Shay. > > > > > > Sorry for long delay, I was out of town. I believe that this problem > > > splits into two: > > > > > > 1. Controlling index-wide terms weight, let's call it IDF. This is the > > > feature with immediate influence on practical applications as it makes > > > introduction of ranking analysis algorithms like SVMRank much easier and > > > strain forward. I don't think however that it should be kept in indexing > > > step. That solution simply breaks the logic: IDF is index-wide and indexing > > > document is "document-wide". I believe that ES needs some terms API that can > > > be used to query for indexed terms information and also for setting their > > > personal weights. > > > One particular problem I can foresee here is text analysis. It's hard to > > > predict how exactly terms from my documents will be transformed during > > > indexing. May be it's worth to return a set of analyzed terms as response > > > for document indexing? > > > > > > 2. Controlling document-wide weights of terms, let's call it TF. Although > > > our conversation started with it, I believe that it much less important and > > > useful then index-wide IDF. Your solution with "the women^1.5 went to the > > > beach^2" is very nice, but I have one concern: how can it play with > > > contradictive inputs? For instance "cat^1 cats^2" in analyzed field. "cat" > > > and "cats" will be analyzed into one single term that will have two weights > > > (1, 2). Out of ideas how to design it in proper way. > > > > > > BTW, is there any way to set all IDF weights for some particular field to > > > 1.0? I found that "omit\_term\_freq\_and\_positions" parameter disables TF, but > > > how to turn off IDF? > > > > > > On Fri, Sep 10, 2010 at 5:12 PM, Shay Banon \< > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote: > > > > > > > Yea, I agree. Though its tricky to expose this functionality nicely... . > > > > Basically, on the indexing side, there is a possibility to control the > > > > weighting of a term for example by adding a payload to it that represent the > > > > score. And then, when searching, using a custom query that takes those > > > > payloads and adds them to the scoring. This means that when indexing, you > > > > will need to provide the weighting (or just the special ones), for example: > > > > "the women^2 went to the beach". This does mean that for each document you > > > > want to change the term weights, it will need to be indexed with a new > > > > value(s), for example: "the women^1.5 went to the beach^2". That format (^) > > > > denoting custom weight is something I invented here for the example. Then, > > > > when searching, those weights will be taken into account using a special > > > > custom\_weight query that will be created for it. Does that sound like > > > > something that will help you? > > > > > > > > On Fri, Sep 10, 2010 at 5:00 PM, Mykhailo Korbakov [rmihael@gmail.com](mailto:rmihael@gmail.com)wrote: > > > > > > > > > Oh I wish that this weird trick would be _my_ wish 🙂 > > > > > I can come up with solution for few of these corrected weights, but > > > > > unfortunately there's a plan for thousands of them. > > > > > > > > > > Don't you think that term weight access (both TF and IDF parts) can be > > > > > useful in ES? Having some way to modify IDF weights is really useful for > > > > > advanced ranking algorithms with implicit feedback processing. As for now we > > > > > have to pass terms weights specified in queries and it doesn't very > > > > > convenient. > > > > > > > > > > May be it is worth to open an issue in GitHub? > > > > > > > > > > On Fri, Sep 10, 2010 at 4:42 PM, Shay Banon \< > > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote: > > > > > > > > > > > No, you can't do that. But you can control it on the search aspect, by > > > > > > constructing a boosted query and filtered by specific docs. This solution > > > > > > really depends on the number of custom weights you wish to set though... . > > > > > > > > > > > > On Fri, Sep 10, 2010 at 4:39 PM, Mykhailo Korbakov \<[rmihael@gmail.com](mailto:rmihael@gmail.com) > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Unfortunately it's not my case. > > > > > > > > > > > > > > My goal is to make changes like this: "set weight of term 'woman' in > > > > > > > document with id 1231231231 to 1.5". I don't think that it's possible with > > > > > > > custom\_score, or at least I'm missing something important about it. > > > > > > > > > > > > > > On Fri, Sep 10, 2010 at 4:21 PM, Shay Banon \< > > > > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote: > > > > > > > > > > > > > > > It really depends on what exactly you are after, but custom\_score > > > > > > > > query might help here: > > > > > > > > [http://www.elasticsearch.com/docs/elasticsearch/rest\_api/query\_dsl/custom\_score\_query/](http://www.elasticsearch.com/docs/elasticsearch/rest_api/query_dsl/custom_score_query/) > > > > > > > > . > > > > > > > > > > > > > > > > On Fri, Sep 10, 2010 at 1:57 AM, rmihael [rmihael@gmail.com](mailto:rmihael@gmail.com) wrote: > > > > > > > > > > > > > > > > > Hi everybody. > > > > > > > > > > > > > > > > > > Is there any way to change term weight in the context of one > > > > > > > > > particular document? Or, other words, to change TF value of > > > > > > > > > specific > > > > > > > > > term for specific document? > > > > > > > > > I need this trick to promote specific documents in the context of > > > > > > > > > predefined queries. May be there's some better way to achieve it? I > > > > > > > > > just can think any other then TF manipulation. > > > > > > > > > > > > > > > > > > Thank you in advance, > > > > > > > > > > > > > > > > > > -- Michael Korbakov ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:18am UTC](https://discuss.elastic.co/t/modification-of-term-weight/3318/11 "2017-07-06T04:18:59Z")