# Optional asciifolding **URL:** \ **Category:** Elasticsearch\ **Created:** [January 28, 2014, 4:00pm UTC](https://discuss.elastic.co/t/optional-asciifolding/15452 "2014-01-28T16:00:55Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [January 28, 2014, 4:00pm UTC](https://discuss.elastic.co/t/optional-asciifolding/15452/1 "2014-01-28T16:00:55Z")
I'm looking to make asciifolding optional in my (English) index. If the user searches without any high ascii characters then I want to match against the folded tokens. If the user searches with high ascii characters then I only want to match the unfolded tokens. Is this possible with Elasticsearch right now? Nik -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3w%2BNHJZQkcCRnEKuowAuObkBTVbHEhnCFpkLH7y0Pa0Q%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3w%2BNHJZQkcCRnEKuowAuObkBTVbHEhnCFpkLH7y0Pa0Q%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Itamar\_Syn\_Hershko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itamar_syn_hershko/32/725_2.png) [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)\ **Post date:** [January 28, 2014, 4:04pm UTC](https://discuss.elastic.co/t/optional-asciifolding/15452/2 "2014-01-28T16:04:53Z")
You will have to use 2 fields, or multiple terms on the same position. In a recent project we found a nice way of dealing with that on the same field, I hope to have a blog post about that soon.. -- Itamar Syn-Hershko [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) Freelance Developer & Consultant Author of RavenDB in Action [http://manning.com/synhershko/](http://manning.com/synhershko/) On Tue, Jan 28, 2014 at 6:00 PM, Nikolas Everett [nik9000@gmail.com](mailto:nik9000@gmail.com) wrote: > I'm looking to make asciifolding optional in my (English) index. If the > user searches without any high ascii characters then I want to match > against the folded tokens. If the user searches with high ascii characters > then I only want to match the unfolded tokens. Is this possible with > Elasticsearch right now? > > Nik > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3w%2BNHJZQkcCRnEKuowAuObkBTVbHEhnCFpkLH7y0Pa0Q%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3w%2BNHJZQkcCRnEKuowAuObkBTVbHEhnCFpkLH7y0Pa0Q%40mail.gmail.com) > . > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuXY6wTDZJNEmwrXN8dRESSYJrKSkcHvSC6KkzYp4TLtg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuXY6wTDZJNEmwrXN8dRESSYJrKSkcHvSC6KkzYp4TLtg%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [January 28, 2014, 4:09pm UTC](https://discuss.elastic.co/t/optional-asciifolding/15452/3 "2014-01-28T16:09:41Z")
I'd prefer multiple terms in the same position if I can get away with it. That way it'd all be configured by the analyzer so it wouldn't add any extra complexity to other languages. It'd take up much less space that way as well. On Tue, Jan 28, 2014 at 11:04 AM, Itamar Syn-Hershko [itamar@code972.com](mailto:itamar@code972.com)wrote: > You will have to use 2 fields, or multiple terms on the same position. In > a recent project we found a nice way of dealing with that on the same > field, I hope to have a blog post about that soon.. > > -- > > Itamar Syn-Hershko > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > Freelance Developer & Consultant > Author of RavenDB in Action [http://manning.com/synhershko/](http://manning.com/synhershko/) > > On Tue, Jan 28, 2014 at 6:00 PM, Nikolas Everett [nik9000@gmail.com](mailto:nik9000@gmail.com)wrote: > > > I'm looking to make asciifolding optional in my (English) index. If the > > user searches without any high ascii characters then I want to match > > against the folded tokens. If the user searches with high ascii characters > > then I only want to match the unfolded tokens. Is this possible with > > Elasticsearch right now? > > > > Nik > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3w%2BNHJZQkcCRnEKuowAuObkBTVbHEhnCFpkLH7y0Pa0Q%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3w%2BNHJZQkcCRnEKuowAuObkBTVbHEhnCFpkLH7y0Pa0Q%40mail.gmail.com) > > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuXY6wTDZJNEmwrXN8dRESSYJrKSkcHvSC6KkzYp4TLtg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuXY6wTDZJNEmwrXN8dRESSYJrKSkcHvSC6KkzYp4TLtg%40mail.gmail.com) > . > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2imi0WzibxZC\_KmeK0J139fR6zjB5H0ij1fdLoxvzJzQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2imi0WzibxZC_KmeK0J139fR6zjB5H0ij1fdLoxvzJzQ%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Itamar\_Syn\_Hershko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itamar_syn_hershko/32/725_2.png) [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)\ **Post date:** [January 28, 2014, 4:20pm UTC](https://discuss.elastic.co/t/optional-asciifolding/15452/4 "2014-01-28T16:20:55Z")
Ok so the idea is you store each term twice - once stemmed (+ ascii folded - whatever) and once just lowercased, and add a character (we used $) to mark that term as the "original". You can see it in action here: [https://github.com/synhershko/elasticsearch-analysis-hebrew/blob/master/src/main/java/com/code972/elasticsearch/analysis/HebrewIndexingAnalyzer.java#L20](https://github.com/synhershko/elasticsearch-analysis-hebrew/blob/master/src/main/java/com/code972/elasticsearch/analysis/HebrewIndexingAnalyzer.java#L20) (warning: plugin still under work, and is using some non-traditional methods to do stuff) There's some details to take into account - like how to search for the original etc, but if you'll look at the code there you'll get an idea of how its done We did that also for non-Hebrew and non-English texts. It works quite nicely, but it doubles the amount of terms in your index. -- Itamar Syn-Hershko [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) Freelance Developer & Consultant Author of RavenDB in Action [http://manning.com/synhershko/](http://manning.com/synhershko/) On Tue, Jan 28, 2014 at 6:09 PM, Nikolas Everett [nik9000@gmail.com](mailto:nik9000@gmail.com) wrote: > I'd prefer multiple terms in the same position if I can get away with it. > That way it'd all be configured by the analyzer so it wouldn't add any > extra complexity to other languages. It'd take up much less space that way > as well. > > On Tue, Jan 28, 2014 at 11:04 AM, Itamar Syn-Hershko [itamar@code972.com](mailto:itamar@code972.com)wrote: > > > You will have to use 2 fields, or multiple terms on the same position. In > > a recent project we found a nice way of dealing with that on the same > > field, I hope to have a blog post about that soon.. > > > > -- > > > > Itamar Syn-Hershko > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > Freelance Developer & Consultant > > Author of RavenDB in Action [http://manning.com/synhershko/](http://manning.com/synhershko/) > > > > On Tue, Jan 28, 2014 at 6:00 PM, Nikolas Everett [nik9000@gmail.com](mailto:nik9000@gmail.com)wrote: > > > > > I'm looking to make asciifolding optional in my (English) index. If > > > the user searches without any high ascii characters then I want to match > > > against the folded tokens. If the user searches with high ascii characters > > > then I only want to match the unfolded tokens. Is this possible with > > > Elasticsearch right now? > > > > > > Nik > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3w%2BNHJZQkcCRnEKuowAuObkBTVbHEhnCFpkLH7y0Pa0Q%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3w%2BNHJZQkcCRnEKuowAuObkBTVbHEhnCFpkLH7y0Pa0Q%40mail.gmail.com) > > > . > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuXY6wTDZJNEmwrXN8dRESSYJrKSkcHvSC6KkzYp4TLtg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuXY6wTDZJNEmwrXN8dRESSYJrKSkcHvSC6KkzYp4TLtg%40mail.gmail.com) > > . > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2imi0WzibxZC\_KmeK0J139fR6zjB5H0ij1fdLoxvzJzQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2imi0WzibxZC_KmeK0J139fR6zjB5H0ij1fdLoxvzJzQ%40mail.gmail.com) > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHTr4Zt2LWsXrQ-fXL49u-azWpGvCqvUh-%2BdN13nYT1SqrOFEQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4Zt2LWsXrQ-fXL49u-azWpGvCqvUh-%2BdN13nYT1SqrOFEQ%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:54am UTC](https://discuss.elastic.co/t/optional-asciifolding/15452/5 "2017-07-06T01:54:21Z")