# Ignore a field in the scoring **URL:** \ **Category:** Elasticsearch\ **Created:** [December 23, 2014, 6:12pm UTC](https://discuss.elastic.co/t/ignore-a-field-in-the-scoring/21373 "2014-12-23T18:12:02Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![roger](https://avatars.discourse-cdn.com/v4/letter/r/c0e974/32.png) [@roger](https://discuss.elastic.co/u/roger)\ **Post date:** [December 23, 2014, 6:12pm UTC](https://discuss.elastic.co/t/ignore-a-field-in-the-scoring/21373/1 "2014-12-23T18:12:02Z")
Hello Our documents have metadata indexed with them, but we don't want the metadata to interfere in the scoring After a user searches for documents, they can bookmark them (what means we add more metadata to the document), then in the next search with the same query the bookmarked document appears in a lower (worse) position Is there a way to completely ignore one or more specific fields in the scoring of every query? as in indexing time or something? Note that we are not using the metadata field in the query, but yet it lowers the score of every query We cannot set the "index" attribute of this field to "no" because we are gonna use it in other queries -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJp2533Rjjec4SwXe\_p-0eHYkkyEegFyP9DUMGQfHhua8ZyMWQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJp2533Rjjec4SwXe_p-0eHYkkyEegFyP9DUMGQfHhua8ZyMWQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [December 26, 2014, 4:59pm UTC](https://discuss.elastic.co/t/ignore-a-field-in-the-scoring/21373/2 "2014-12-26T16:59:33Z")
Use the field in a filter and not part of the query. Is this field free text? Ivan On Dec 23, 2014 9:12 PM, "Roger de Cordova Farias" \< roger.farias@fontec.inf.br\> wrote: > Hello > > Our documents have metadata indexed with them, but we don't want the > metadata to interfere in the scoring > > After a user searches for documents, they can bookmark them (what means we > add more metadata to the document), then in the next search with the same > query the bookmarked document appears in a lower (worse) position > > Is there a way to completely ignore one or more specific fields in the > scoring of every query? as in indexing time or something? > > Note that we are not using the metadata field in the query, but yet it > lowers the score of every query > > We cannot set the "index" attribute of this field to "no" because we are > gonna use it in other queries > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAJp2533Rjjec4SwXe\_p-0eHYkkyEegFyP9DUMGQfHhua8ZyMWQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJp2533Rjjec4SwXe_p-0eHYkkyEegFyP9DUMGQfHhua8ZyMWQ%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAJp2533Rjjec4SwXe\_p-0eHYkkyEegFyP9DUMGQfHhua8ZyMWQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAJp2533Rjjec4SwXe_p-0eHYkkyEegFyP9DUMGQfHhua8ZyMWQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAKsYquQJMbfztJ%2Ba2\_jpi-fVG%3DvcnXYHS-7bKvaOX4hA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAKsYquQJMbfztJ%2Ba2_jpi-fVG%3DvcnXYHS-7bKvaOX4hA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![softwaredoug](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/softwaredoug/32/22681_2.png) [@softwaredoug](https://discuss.elastic.co/u/softwaredoug)\ **Post date:** [December 26, 2014, 7:34pm UTC](https://discuss.elastic.co/t/ignore-a-field-in-the-scoring/21373/3 "2014-12-26T19:34:31Z")
Are you querying the \_all field? How are you doing your searches? > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. The \_all field receives a copy of every field you index, so adding data here could impact scores regardless of the source field. Otherwise, fields are scored independently before being put together by other queries like boolean queries or dismax. Are you using boolean/dismax/etc over multiple fields? -Doug On Fri, Dec 26, 2014 at 11:59 AM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote: > Use the field in a filter and not part of the query. Is this field free > text? > > Ivan > On Dec 23, 2014 9:12 PM, "Roger de Cordova Farias" \< > roger.farias@fontec.inf.br\> wrote: > > > Hello > > > > Our documents have metadata indexed with them, but we don't want the > > metadata to interfere in the scoring > > > > After a user searches for documents, they can bookmark them (what means > > we add more metadata to the document), then in the next search with the > > same query the bookmarked document appears in a lower (worse) position > > > > Is there a way to completely ignore one or more specific fields in the > > scoring of every query? as in indexing time or something? > > > > Note that we are not using the metadata field in the query, but yet it > > lowers the score of every query > > > > We cannot set the "index" attribute of this field to "no" because we are > > gonna use it in other queries > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAJp2533Rjjec4SwXe\_p-0eHYkkyEegFyP9DUMGQfHhua8ZyMWQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJp2533Rjjec4SwXe_p-0eHYkkyEegFyP9DUMGQfHhua8ZyMWQ%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CAJp2533Rjjec4SwXe\_p-0eHYkkyEegFyP9DUMGQfHhua8ZyMWQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAJp2533Rjjec4SwXe_p-0eHYkkyEegFyP9DUMGQfHhua8ZyMWQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAKsYquQJMbfztJ%2Ba2\_jpi-fVG%3DvcnXYHS-7bKvaOX4hA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAKsYquQJMbfztJ%2Ba2_jpi-fVG%3DvcnXYHS-7bKvaOX4hA%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAKsYquQJMbfztJ%2Ba2\_jpi-fVG%3DvcnXYHS-7bKvaOX4hA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAKsYquQJMbfztJ%2Ba2_jpi-fVG%3DvcnXYHS-7bKvaOX4hA%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- Doug Turnbull Search & Big Data Architect OpenSource Connections [http://o19s.com](http://o19s.com) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CALG6HL9ND\_SWteSetZL9059WyGRZvJrO2k4PQ9FQ1zUFhjbsxw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CALG6HL9ND_SWteSetZL9059WyGRZvJrO2k4PQ9FQ1zUFhjbsxw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![roger](https://avatars.discourse-cdn.com/v4/letter/r/c0e974/32.png) [@roger](https://discuss.elastic.co/u/roger)\ **Post date:** [January 5, 2015, 3:35pm UTC](https://discuss.elastic.co/t/ignore-a-field-in-the-scoring/21373/4 "2015-01-05T15:35:12Z")
The added field is an array of Integers, but we are not using it in the query at all We are not querying the \_all field, it is disabled in our type mapping Our query is something like this: { > "query": { > "query\_string": { > "fields": [ > "name" > ], > "query": "roger" > } > } > } I ran this query. In the first result, I added a new field called "bookmarked\_by" with a numeric value. Then I ran the same query again. The document in which I added the new field is no longer the first result 2014-12-26 17:34 GMT-02:00 Doug Turnbull \< [dturnbull@opensourceconnections.com](mailto:dturnbull@opensourceconnections.com)\>: > Are you querying the \_all field? How are you doing your searches? > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-all-field.html) > > The \_all field receives a copy of every field you index, so adding data > here could impact scores regardless of the source field. > > Otherwise, fields are scored independently before being put together by > other queries like boolean queries or dismax. Are you using > boolean/dismax/etc over multiple fields? > > -Doug > > On Fri, Dec 26, 2014 at 11:59 AM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote: > > > Use the field in a filter and not part of the query. Is this field free > > text? > > > > Ivan > > On Dec 23, 2014 9:12 PM, "Roger de Cordova Farias" \< > > roger.farias@fontec.inf.br\> wrote: > > > > > Hello > > > > > > Our documents have metadata indexed with them, but we don't want the > > > metadata to interfere in the scoring > > > > > > After a user searches for documents, they can bookmark them (what means > > > we add more metadata to the document), then in the next search with the > > > same query the bookmarked document appears in a lower (worse) position > > > > > > Is there a way to completely ignore one or more specific fields in the > > > scoring of every query? as in indexing time or something? > > > > > > Note that we are not using the metadata field in the query, but yet it > > > lowers the score of every query > > > > > > We cannot set the "index" attribute of this field to "no" because we are > > > gonna use it in other queries > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/CAJp2533Rjjec4SwXe\_p-0eHYkkyEegFyP9DUMGQfHhua8ZyMWQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJp2533Rjjec4SwXe_p-0eHYkkyEegFyP9DUMGQfHhua8ZyMWQ%40mail.gmail.com) > > > [https://groups.google.com/d/msgid/elasticsearch/CAJp2533Rjjec4SwXe\_p-0eHYkkyEegFyP9DUMGQfHhua8ZyMWQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAJp2533Rjjec4SwXe_p-0eHYkkyEegFyP9DUMGQfHhua8ZyMWQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAKsYquQJMbfztJ%2Ba2\_jpi-fVG%3DvcnXYHS-7bKvaOX4hA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAKsYquQJMbfztJ%2Ba2_jpi-fVG%3DvcnXYHS-7bKvaOX4hA%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAKsYquQJMbfztJ%2Ba2\_jpi-fVG%3DvcnXYHS-7bKvaOX4hA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAKsYquQJMbfztJ%2Ba2_jpi-fVG%3DvcnXYHS-7bKvaOX4hA%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > Doug Turnbull > Search & Big Data Architect > OpenSource Connections [http://o19s.com](http://o19s.com) > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CALG6HL9ND\_SWteSetZL9059WyGRZvJrO2k4PQ9FQ1zUFhjbsxw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CALG6HL9ND_SWteSetZL9059WyGRZvJrO2k4PQ9FQ1zUFhjbsxw%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CALG6HL9ND\_SWteSetZL9059WyGRZvJrO2k4PQ9FQ1zUFhjbsxw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CALG6HL9ND_SWteSetZL9059WyGRZvJrO2k4PQ9FQ1zUFhjbsxw%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJp2533UjpAz2dvNitdD-%3DaoXL9rrkZdd%3DzC3LZz8xWYvBAoFQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJp2533UjpAz2dvNitdD-%3DaoXL9rrkZdd%3DzC3LZz8xWYvBAoFQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![roger](https://avatars.discourse-cdn.com/v4/letter/r/c0e974/32.png) [@roger](https://discuss.elastic.co/u/roger)\ **Post date:** [January 5, 2015, 6:16pm UTC](https://discuss.elastic.co/t/ignore-a-field-in-the-scoring/21373/5 "2015-01-05T18:16:23Z")
Now I ran the query with explain = true. The results are the following: _Explain before the update:_ > ``` > "details": [ > { > "value": 5.752348, > "description": "fieldWeight in 424, product of:", > "details": [ > { > "value": 1, > "description": "tf(freq=1.0), with freq of:", > "details": [ > { > "value": 1, > "description": "termFreq=1.0" > } > ] > }, > { > "value": 9.203756, > "description": "idf(docFreq=201, maxDocs=738240)" > }, > { > "value": 0.625, > "description": "fieldNorm(doc=424)" > } > ] > } > ] > > ``` _Update script (scriptLang = groovy, profileId = 1):_ if (ctx.\_source.bookmarked\_by == null) { > ``` > ctx._source.bookmarked_by = [profileId] > > ``` > > } else if (ctx.\_source.bookmarked\_by.contains(profileId)) { > ctx.op = "none" > } else { > ctx.\_source.bookmarked\_by += profileId > } _Explain after the update:_ ``` "details": [ ``` > ``` > { > "value": 5.749262, > "description": "fieldWeight in 0, product of:", > "details": [ > { > "value": 1, > "description": "tf(freq=1.0), with freq of:", > "details": [ > { > "value": 1, > "description": "termFreq=1.0" > } > ] > }, > { > "value": 9.198819, > "description": "idf(docFreq=202, maxDocs=738241)" > }, > { > "value": 0.625, > "description": "fieldNorm(doc=0)" > } > ] > } > ] > > ``` - Query used with the explain:\* { > "query": { > "query\_string": { > "fields": [ > "name" > ], > "query": "roger" > } > } > } The inverse document frequency (idf) is changed after adding a new field that is not used in the query. Also, it changed the "fieldWeight in 424" and "fieldNorm(doc=424)" to "fieldWeight in 0" and "fieldNorm(doc=0)" (idk if it changes something) Can someone help me on how to not change the score of the document after running the update? Note that the update creates a new field if it was not found (== null), but this field is not used in the query 2015-01-05 13:35 GMT-02:00 Roger de Cordova Farias \< roger.farias@fontec.inf.br\>: > The added field is an array of Integers, but we are not using it in the > query at all > > We are not querying the \_all field, it is disabled in our type mapping > > Our query is something like this: > > { > > > "query": { > > "query\_string": { > > "fields": [ > > "name" > > ], > > "query": "roger" > > } > > } > > } > > I ran this query. In the first result, I added a new field called > "bookmarked\_by" with a numeric value. Then I ran the same query again. The > document in which I added the new field is no longer the first result > > 2014-12-26 17:34 GMT-02:00 Doug Turnbull \< > [dturnbull@opensourceconnections.com](mailto:dturnbull@opensourceconnections.com)\>: > > Are you querying the \_all field? How are you doing your searches? > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-all-field.html) > > > > The \_all field receives a copy of every field you index, so adding data > > here could impact scores regardless of the source field. > > > > Otherwise, fields are scored independently before being put together by > > other queries like boolean queries or dismax. Are you using > > boolean/dismax/etc over multiple fields? > > > > -Doug > > > > On Fri, Dec 26, 2014 at 11:59 AM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote: > > > > > Use the field in a filter and not part of the query. Is this field free > > > text? > > > > > > Ivan > > > On Dec 23, 2014 9:12 PM, "Roger de Cordova Farias" \< > > > roger.farias@fontec.inf.br\> wrote: > > > > > > > Hello > > > > > > > > Our documents have metadata indexed with them, but we don't want the > > > > metadata to interfere in the scoring > > > > > > > > After a user searches for documents, they can bookmark them (what means > > > > we add more metadata to the document), then in the next search with the > > > > same query the bookmarked document appears in a lower (worse) position > > > > > > > > Is there a way to completely ignore one or more specific fields in the > > > > scoring of every query? as in indexing time or something? > > > > > > > > Note that we are not using the metadata field in the query, but yet it > > > > lowers the score of every query > > > > > > > > We cannot set the "index" attribute of this field to "no" because we > > > > are gonna use it in other queries > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/CAJp2533Rjjec4SwXe\_p-0eHYkkyEegFyP9DUMGQfHhua8ZyMWQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJp2533Rjjec4SwXe_p-0eHYkkyEegFyP9DUMGQfHhua8ZyMWQ%40mail.gmail.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/CAJp2533Rjjec4SwXe\_p-0eHYkkyEegFyP9DUMGQfHhua8ZyMWQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAJp2533Rjjec4SwXe_p-0eHYkkyEegFyP9DUMGQfHhua8ZyMWQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAKsYquQJMbfztJ%2Ba2\_jpi-fVG%3DvcnXYHS-7bKvaOX4hA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAKsYquQJMbfztJ%2Ba2_jpi-fVG%3DvcnXYHS-7bKvaOX4hA%40mail.gmail.com) > > > [https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAKsYquQJMbfztJ%2Ba2\_jpi-fVG%3DvcnXYHS-7bKvaOX4hA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAKsYquQJMbfztJ%2Ba2_jpi-fVG%3DvcnXYHS-7bKvaOX4hA%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > Doug Turnbull > > Search & Big Data Architect > > OpenSource Connections [http://o19s.com](http://o19s.com) > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CALG6HL9ND\_SWteSetZL9059WyGRZvJrO2k4PQ9FQ1zUFhjbsxw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CALG6HL9ND_SWteSetZL9059WyGRZvJrO2k4PQ9FQ1zUFhjbsxw%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CALG6HL9ND\_SWteSetZL9059WyGRZvJrO2k4PQ9FQ1zUFhjbsxw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CALG6HL9ND_SWteSetZL9059WyGRZvJrO2k4PQ9FQ1zUFhjbsxw%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJp2531gd\_T5rRm2a4JsiGGwd2kQrmFxgiW0iAFMceL2PyzUWQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJp2531gd_T5rRm2a4JsiGGwd2kQrmFxgiW0iAFMceL2PyzUWQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![masaru](https://avatars.discourse-cdn.com/v4/letter/m/a9a28c/32.png) [@masaru](https://discuss.elastic.co/u/masaru)\ **Post date:** [January 7, 2015, 3:53am UTC](https://discuss.elastic.co/t/ignore-a-field-in-the-scoring/21373/6 "2015-01-07T03:53:13Z")
Hi, Update is delete and add. I mean, instead of updating existing document, it deletes it and adds it as new document. And those deleted documents are just marked as deleted and aren’t actually removed from index until the segment merge. IDF doesn’t take those deleted-but-not-removed document into account (it counts those documents). That’s the reason you see different IDF score (you see both maxDocs and docFreq are incremented). Regarding 424 v.s. 0, the document had ID 424 (lucene’s internal ID). But when the document is updated (delete + add), it got new ID 0 in new segment. So, I think it’s not possible to keep score when you update documents. You can run optimise with max\_num\_segments=1 every time you update documents but it’s not practical (and until optimise is done, you see different score) Masaru -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/etPan.54acade5.625558ec.13b%40citra.local](https://groups.google.com/d/msgid/elasticsearch/etPan.54acade5.625558ec.13b%40citra.local). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![roger](https://avatars.discourse-cdn.com/v4/letter/r/c0e974/32.png) [@roger](https://discuss.elastic.co/u/roger)\ **Post date:** [January 7, 2015, 3:01pm UTC](https://discuss.elastic.co/t/ignore-a-field-in-the-scoring/21373/7 "2015-01-07T15:01:40Z")
Thank you for your explanation Do you know if it is a bug of intended behavior? I don't think deleted (marked as deleted) docs should be used at all 2015-01-07 1:53 GMT-02:00 Masaru Hasegawa [haniomasali@gmail.com](mailto:haniomasali@gmail.com): > Hi, > > Update is delete and add. I mean, instead of updating existing document, > it deletes it and adds it as new document. > And those deleted documents are just marked as deleted and aren’t actually > removed from index until the segment merge. > > IDF doesn’t take those deleted-but-not-removed document into account (it > counts those documents). > That’s the reason you see different IDF score (you see both maxDocs and > docFreq are incremented). > > Regarding 424 v.s. 0, the document had ID 424 (lucene’s internal ID). But > when the document is updated (delete + add), it got new ID 0 in new segment. > > So, I think it’s not possible to keep score when you update documents. > You can run optimise with max\_num\_segments=1 every time you update > documents but it’s not practical (and until optimise is done, you see > different score) > > Masaru > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/etPan.54acade5.625558ec.13b%40citra.local](https://groups.google.com/d/msgid/elasticsearch/etPan.54acade5.625558ec.13b%40citra.local) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJp2531fazjRDeFMmWLVuoCtCUtbCUMv841O%2BZoFpMJBdcjRDA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJp2531fazjRDeFMmWLVuoCtCUtbCUMv841O%2BZoFpMJBdcjRDA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![masaru](https://avatars.discourse-cdn.com/v4/letter/m/a9a28c/32.png) [@masaru](https://discuss.elastic.co/u/masaru)\ **Post date:** [January 8, 2015, 6:35am UTC](https://discuss.elastic.co/t/ignore-a-field-in-the-scoring/21373/8 "2015-01-08T06:35:27Z")
Hi, ## I believe it's intended according to [TFIDFSimilarity (Lucene 4.10.2 API)](https://lucene.apache.org/core/4_10_2/core/org/apache/lucene/search/similarities/TFIDFSimilarity.html) . It says: ## Note that CollectionStatistics.maxDoc() is used instead of IndexReader#numDocs() because also TermStatistics.docFreq() is used, and when the latter is inaccurate, so is CollectionStatistics.maxDoc(), and in the same direction. In addition, CollectionStatistics.maxDoc() is more efficient to compute Masaru On Thu, Jan 8, 2015 at 12:01 AM, Roger de Cordova Farias \< roger.farias@fontec.inf.br\> wrote: > Thank you for your explanation > > Do you know if it is a bug of intended behavior? > > I don't think deleted (marked as deleted) docs should be used at all > > 2015-01-07 1:53 GMT-02:00 Masaru Hasegawa [haniomasali@gmail.com](mailto:haniomasali@gmail.com): > > > Hi, > > > > Update is delete and add. I mean, instead of updating existing document, > > it deletes it and adds it as new document. > > And those deleted documents are just marked as deleted and aren’t > > actually removed from index until the segment merge. > > > > IDF doesn’t take those deleted-but-not-removed document into account (it > > counts those documents). > > That’s the reason you see different IDF score (you see both maxDocs and > > docFreq are incremented). > > > > Regarding 424 v.s. 0, the document had ID 424 (lucene’s internal ID). But > > when the document is updated (delete + add), it got new ID 0 in new segment. > > > > So, I think it’s not possible to keep score when you update documents. > > You can run optimise with max\_num\_segments=1 every time you update > > documents but it’s not practical (and until optimise is done, you see > > different score) > > > > Masaru > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/etPan.54acade5.625558ec.13b%40citra.local](https://groups.google.com/d/msgid/elasticsearch/etPan.54acade5.625558ec.13b%40citra.local) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAJp2531fazjRDeFMmWLVuoCtCUtbCUMv841O%2BZoFpMJBdcjRDA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJp2531fazjRDeFMmWLVuoCtCUtbCUMv841O%2BZoFpMJBdcjRDA%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAJp2531fazjRDeFMmWLVuoCtCUtbCUMv841O%2BZoFpMJBdcjRDA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAJp2531fazjRDeFMmWLVuoCtCUtbCUMv841O%2BZoFpMJBdcjRDA%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAGmu3c1rWBCuaLrwHY818sy%2BcM6wEYzNivcFMjzbqupW\_7paAw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGmu3c1rWBCuaLrwHY818sy%2BcM6wEYzNivcFMjzbqupW_7paAw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![roger](https://avatars.discourse-cdn.com/v4/letter/r/c0e974/32.png) [@roger](https://discuss.elastic.co/u/roger)\ **Post date:** [January 8, 2015, 2:11pm UTC](https://discuss.elastic.co/t/ignore-a-field-in-the-scoring/21373/9 "2015-01-08T14:11:57Z")
Thank you very much 2015-01-08 4:35 GMT-02:00 Masaru Hasegawa [haniomasali@gmail.com](mailto:haniomasali@gmail.com): > Hi, > > ## I believe it's intended according to [TFIDFSimilarity (Lucene 4.10.2 API)](https://lucene.apache.org/core/4_10_2/core/org/apache/lucene/search/similarities/TFIDFSimilarity.html) . It says: > > ## Note that CollectionStatistics.maxDoc() is used instead of IndexReader#numDocs() because also TermStatistics.docFreq() is used, and when the latter is inaccurate, so is CollectionStatistics.maxDoc(), and in the same direction. In addition, CollectionStatistics.maxDoc() is more efficient to compute > > Masaru > > On Thu, Jan 8, 2015 at 12:01 AM, Roger de Cordova Farias \< > roger.farias@fontec.inf.br\> wrote: > > > Thank you for your explanation > > > > Do you know if it is a bug of intended behavior? > > > > I don't think deleted (marked as deleted) docs should be used at all > > > > 2015-01-07 1:53 GMT-02:00 Masaru Hasegawa [haniomasali@gmail.com](mailto:haniomasali@gmail.com): > > > > > Hi, > > > > > > Update is delete and add. I mean, instead of updating existing document, > > > it deletes it and adds it as new document. > > > And those deleted documents are just marked as deleted and aren’t > > > actually removed from index until the segment merge. > > > > > > IDF doesn’t take those deleted-but-not-removed document into account (it > > > counts those documents). > > > That’s the reason you see different IDF score (you see both maxDocs and > > > docFreq are incremented). > > > > > > Regarding 424 v.s. 0, the document had ID 424 (lucene’s internal ID). > > > But when the document is updated (delete + add), it got new ID 0 in new > > > segment. > > > > > > So, I think it’s not possible to keep score when you update documents. > > > You can run optimise with max\_num\_segments=1 every time you update > > > documents but it’s not practical (and until optimise is done, you see > > > different score) > > > > > > Masaru > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/etPan.54acade5.625558ec.13b%40citra.local](https://groups.google.com/d/msgid/elasticsearch/etPan.54acade5.625558ec.13b%40citra.local) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAJp2531fazjRDeFMmWLVuoCtCUtbCUMv841O%2BZoFpMJBdcjRDA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJp2531fazjRDeFMmWLVuoCtCUtbCUMv841O%2BZoFpMJBdcjRDA%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CAJp2531fazjRDeFMmWLVuoCtCUtbCUMv841O%2BZoFpMJBdcjRDA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAJp2531fazjRDeFMmWLVuoCtCUtbCUMv841O%2BZoFpMJBdcjRDA%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAGmu3c1rWBCuaLrwHY818sy%2BcM6wEYzNivcFMjzbqupW\_7paAw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGmu3c1rWBCuaLrwHY818sy%2BcM6wEYzNivcFMjzbqupW_7paAw%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAGmu3c1rWBCuaLrwHY818sy%2BcM6wEYzNivcFMjzbqupW\_7paAw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAGmu3c1rWBCuaLrwHY818sy%2BcM6wEYzNivcFMjzbqupW_7paAw%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJp2533-8TBoyPmfpqj12T\_TVb4z%2BrgLKqtuOxRfReajti7WfA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJp2533-8TBoyPmfpqj12T_TVb4z%2BrgLKqtuOxRfReajti7WfA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:40am UTC](https://discuss.elastic.co/t/ignore-a-field-in-the-scoring/21373/10 "2017-07-06T00:40:01Z")