# Removing non-unique tokens during indexing **URL:** \ **Category:** Elasticsearch\ **Created:** [August 22, 2013, 3:47pm UTC](https://discuss.elastic.co/t/removing-non-unique-tokens-during-indexing/13302 "2013-08-22T15:47:14Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![maciej](https://avatars.discourse-cdn.com/v4/letter/m/f9ae1b/32.png) [@maciej](https://discuss.elastic.co/u/maciej)\ **Post date:** [August 22, 2013, 3:47pm UTC](https://discuss.elastic.co/t/removing-non-unique-tokens-during-indexing/13302/1 "2013-08-22T15:47:14Z")
Hi I came up with idea of indexing only document-wide unique tokens in a field. So for example having this document: { 'category': 'wine', 'description': 'bottle of wine from Marlow brewery' } i'd like to remove all tokens in category field from description - making it better suited for search. Which means i need to find a place that has tookens for both fields and remove some of them. Is there there an easy way to achieve that during indexing on ES side? Thanks for any advice -- [fiedzia@gmail.com](mailto:fiedzia@gmail.com) Maciej Dziardziel -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [August 22, 2013, 4:09pm UTC](https://discuss.elastic.co/t/removing-non-unique-tokens-during-indexing/13302/2 "2013-08-22T16:09:16Z")
You can add both fields to the \_all field [1] and then apply an analyzer with a unique token filter [2] to the \_all field. Make sure to exclude other fields from \_all. You can achieve something similar to the \_all field with multi field merging [3]. [1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/mapping/all-field/) [2] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/analysis/unique-tokenfilter/) [3] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/mapping/multi-field-type/) Cheers, Ivan On Thu, Aug 22, 2013 at 8:47 AM, [maciej@ly.st](mailto:maciej@ly.st) wrote: > Hi > > I came up with idea of indexing only document-wide unique tokens in a > field. So for example having this document: > > { > 'category': 'wine', > 'description': 'bottle of wine from Marlow brewery' > } > > i'd like to remove all tokens in category field from description - making > it better suited for search. > > Which means i need to find a place that has tookens for both fields and > remove some of them. > Is there there an easy way to achieve that during indexing on ES side? > > Thanks for any advice > > -- > [fiedzia@gmail.com](mailto:fiedzia@gmail.com) > Maciej Dziardziel > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![maciej](https://avatars.discourse-cdn.com/v4/letter/m/f9ae1b/32.png) [@maciej](https://discuss.elastic.co/u/maciej)\ **Post date:** [August 23, 2013, 11:40am UTC](https://discuss.elastic.co/t/removing-non-unique-tokens-during-indexing/13302/3 "2013-08-23T11:40:04Z")
Thanks - this may be useful, but i need more flexibility. multifield can add fields, but i want to subtract them. Could plugin do that? On Thursday, August 22, 2013 5:09:16 PM UTC+1, Ivan Brusic wrote: > You can add both fields to the \_all field [1] and then apply an analyzer > with a unique token filter [2] to the \_all field. Make sure to exclude > other fields from \_all. You can achieve something similar to the \_all field > with multi field merging [3]. > > [1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/mapping/all-field/) > [2] > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/analysis/unique-tokenfilter/) > [3] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/mapping/multi-field-type/) > > Cheers, > Ivan > > On Thu, Aug 22, 2013 at 8:47 AM, \\> wrote: > > > On Thu, Aug 22, 2013 at 8:47 AM, \\> wrote: > > > > Hi > > > > I came up with idea of indexing only document-wide unique tokens in a > > field. So for example having this document: > > > > { > > 'category': 'wine', > > 'description': 'bottle of wine from Marlow brewery' > > } > > > > i'd like to remove all tokens in category field from description - making > > it better suited for search. > > > > Which means i need to find a place that has tookens for both fields and > > remove some of them. > > Is there there an easy way to achieve that during indexing on ES side? > > > > Thanks for any advice > > > > -- > > [fie...@gmail.com](mailto:fie...@gmail.com) \ > > Maciej Dziardziel > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Maciej\_Dziardziel](https://avatars.discourse-cdn.com/v4/letter/m/d9b06d/32.png) [@Maciej\_Dziardziel](https://discuss.elastic.co/u/Maciej_Dziardziel)\ **Post date:** [August 24, 2013, 10:46am UTC](https://discuss.elastic.co/t/removing-non-unique-tokens-during-indexing/13302/4 "2013-08-24T10:46:54Z")
Turned out, stop filter did the trick. On Friday, August 23, 2013 12:40:04 PM UTC+1, mac...@ly.st wrote: > Thanks - this may be useful, but i need more flexibility. multifield can > add fields, but i want to subtract them. Could plugin do that? > > On Thursday, August 22, 2013 5:09:16 PM UTC+1, Ivan Brusic wrote: > > > You can add both fields to the \_all field [1] and then apply an analyzer > > with a unique token filter [2] to the \_all field. Make sure to exclude > > other fields from \_all. You can achieve something similar to the \_all field > > with multi field merging [3]. > > > > [1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/mapping/all-field/) > > [2] > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/analysis/unique-tokenfilter/) > > [3] > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/mapping/multi-field-type/) > > > > Cheers, > > Ivan > > > > On Thu, Aug 22, 2013 at 8:47 AM, [mac...@ly.st](mailto:mac...@ly.st) wrote: > > > > > On Thu, Aug 22, 2013 at 8:47 AM, [mac...@ly.st](mailto:mac...@ly.st) wrote: > > > > > > Hi > > > > > > I came up with idea of indexing only document-wide unique tokens in a > > > field. So for example having this document: > > > > > > { > > > 'category': 'wine', > > > 'description': 'bottle of wine from Marlow brewery' > > > } > > > > > > i'd like to remove all tokens in category field from description - > > > making it better suited for search. > > > > > > Which means i need to find a place that has tookens for both fields and > > > remove some of them. > > > Is there there an easy way to achieve that during indexing on ES side? > > > > > > Thanks for any advice > > > > > > -- > > > [fie...@gmail.com](mailto:fie...@gmail.com) > > > Maciej Dziardziel > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:19am UTC](https://discuss.elastic.co/t/removing-non-unique-tokens-during-indexing/13302/5 "2017-07-06T02:19:56Z")