# EdgeNgram tokenizer: Get only high quality hits? **URL:** \ **Category:** Elasticsearch\ **Created:** [January 4, 2014, 5:39am UTC](https://discuss.elastic.co/t/edgengram-tokenizer-get-only-high-quality-hits/15105 "2014-01-04T05:39:33Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Karan\_Verma](https://avatars.discourse-cdn.com/v4/letter/k/858c86/32.png) [@Karan\_Verma](https://discuss.elastic.co/u/Karan_Verma)\ **Post date:** [January 4, 2014, 5:39am UTC](https://discuss.elastic.co/t/edgengram-tokenizer-get-only-high-quality-hits/15105/1 "2014-01-04T05:39:33Z")
Hi I implemented a custom filter which uses the EdgeGram tokenizer. The problem I face is that whether I search for something relevant or total garbage I get a large number of hits. I suspect that this is due that fact that I'm using an EdgeNgram tokenizer. How can I ensure the quality of the search results and have ES return no matches for low quality results? Does the fact that elastic search returns a large number of hits (~100,000) mean that there's probably something wrong in how I'm constructing the query? And does returning this many results affect query time substantially? Best, Karan -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/8bf474c2-7559-4fe4-b925-a62aedf3f09b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8bf474c2-7559-4fe4-b925-a62aedf3f09b%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [January 9, 2014, 7:46am UTC](https://discuss.elastic.co/t/edgengram-tokenizer-get-only-high-quality-hits/15105/2 "2014-01-09T07:46:05Z")
Hey, answering this question is sort of philosophical 🙂 First, noone knows the requirements of your application. There might be applications where it is inacceptable to not return any hits, so people like to rather return 100k hits instead of none. Most of the time this is not the case though. Second, those 100k hits might be valid. Maybe you just used a generic search term. Hard to tell from the outside, if your search quality is good or bad. Judging from your comment, it sounds different however. So, what can be done about it? You should try and use the analyze API, which helps you, how the content of a field is acutally broken down and stored in the index. This automatically helps you to understand, why a certain term might match on this document. Also the Explain API may be used for this. See > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. If you want to use a nice graphical frontend for the analyze API, try the inquisitor plugin. In your setup another strategy might be to try another search first, and only if that one does not match, you could use the (edge)ngram one, which is most likely to return more hits. Also, I guess there was a reason, why you chose that configuration in the first place. With a couple of examples, we might help to explain, why certain documents are actually being returned. A reproducible gist for others might help here. Hope this helps as a start to dive deeper into your issue. --Alex On Sat, Jan 4, 2014 at 6:39 AM, Karan Verma [karan@healthtap.com](mailto:karan@healthtap.com) wrote: > Hi > > I implemented a custom filter which uses the EdgeGram tokenizer. The > problem I face is that whether I search for something relevant or total > garbage I get a large number of hits. I suspect that this is due that fact > that I'm using an EdgeNgram tokenizer. How can I ensure the quality of the > search results and have ES return no matches for low quality results? > > Does the fact that Elasticsearch returns a large number of hits > (~100,000) mean that there's probably something wrong in how I'm > constructing the query? And does returning this many results affect query > time substantially? > > Best, > Karan > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/8bf474c2-7559-4fe4-b925-a62aedf3f09b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8bf474c2-7559-4fe4-b925-a62aedf3f09b%40googlegroups.com) > . > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAGCwEM803SPZOcUTtd9VzAV04fwmA1%3D6W6hdKbA3iBEXEc7T%2BQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCwEM803SPZOcUTtd9VzAV04fwmA1%3D6W6hdKbA3iBEXEc7T%2BQ%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:57am UTC](https://discuss.elastic.co/t/edgengram-tokenizer-get-only-high-quality-hits/15105/3 "2017-07-06T01:57:48Z")
Closed on July 6, 2017, 1:57am UTC