# Alternative to ngram tokenizer and token filter. Substring issue **URL:** \ **Category:** Elasticsearch\ **Created:** [June 20, 2016, 6:14pm UTC](https://discuss.elastic.co/t/alternative-to-ngram-tokenizer-and-token-filter-substring-issue/53354 "2016-06-20T18:14:41Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![lantern77](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lantern77/32/22043_2.png) [@lantern77](https://discuss.elastic.co/u/lantern77)\ **Post date:** [June 20, 2016, 6:14pm UTC](https://discuss.elastic.co/t/alternative-to-ngram-tokenizer-and-token-filter-substring-issue/53354/1 "2016-06-20T18:14:41Z")
Hey guys, Currently I am looking for a means to search for a substring inside a phrase or text. I know that the two most common methods are is to use: 1. ngram tokenizer/ token filter. 2. Wild card search. However, both suffer from a trade off. 1 the ngram tokenizer suffers from too much data overhead and its something we can't afford (note we would like ngrams of a large range to have more accuracy). While 2. the wild card search is unfortunately too slow, especially for sub-strings that appear often in searches and searches that begin with the \* itself. I was wondering if there was a one size fits all or a jack of all trades solution here. Where not too much memory used yet still remaining fast and accurate. Regards Peter. P.S I have thought about using the reverse token filter and generating two fields, and making two term queries to cover strings that start with and end with . However that solution did not cover, sub-strings. Strings in the middle of the term we want. ---
**Author:** ![softwaredoug](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/softwaredoug/32/22681_2.png) [@softwaredoug](https://discuss.elastic.co/u/softwaredoug)\ **Post date:** [June 20, 2016, 7:16pm UTC](https://discuss.elastic.co/t/alternative-to-ngram-tokenizer-and-token-filter-substring-issue/53354/2 "2016-06-20T19:16:59Z")
Are you trying to search for arbitrary substrings in terms? What sorts of sample queries would you have Would you expect a query "an" to match "banana"? Or are you looking for some kind of typo tolerance -- single mistakes etc? ---
**Author:** ![lantern77](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lantern77/32/22043_2.png) [@lantern77](https://discuss.elastic.co/u/lantern77)\ **Post date:** [June 20, 2016, 7:36pm UTC](https://discuss.elastic.co/t/alternative-to-ngram-tokenizer-and-token-filter-substring-issue/53354/3 "2016-06-20T19:36:55Z")
Yes I am trying to search for arbitrary sub-strings. So "an" in banana should work. ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [June 20, 2016, 7:52pm UTC](https://discuss.elastic.co/t/alternative-to-ngram-tokenizer-and-token-filter-substring-issue/53354/4 "2016-06-20T19:52:49Z")
> [@lantern77](#): > > 1. the wild card search is unfortunately too slow, especially for sub-strings that appear often in searches and searches that begin with the \* itself. Reverse [edgeNGram](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-edgengram-tokenizer.html) on the analyze side and prefix queries on the search side offers a decent tradeoff I think. I expect the space usage to be much, much, much better than ngrams and the performance to be significantly better than wildcard search. ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [June 20, 2016, 7:53pm UTC](https://discuss.elastic.co/t/alternative-to-ngram-tokenizer-and-token-filter-substring-issue/53354/5 "2016-06-20T19:53:12Z")
I'm just shooting from the hip here. I haven't tested it. But I expect it'll work. ---
**Author:** ![lantern77](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lantern77/32/22043_2.png) [@lantern77](https://discuss.elastic.co/u/lantern77)\ **Post date:** [June 20, 2016, 7:54pm UTC](https://discuss.elastic.co/t/alternative-to-ngram-tokenizer-and-token-filter-substring-issue/53354/6 "2016-06-20T19:54:52Z")
Thanks I'll definitely try it out, bit worried about accuracy but that's something I will trade off . ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:42pm UTC](https://discuss.elastic.co/t/alternative-to-ngram-tokenizer-and-token-filter-substring-issue/53354/7 "2017-07-05T22:42:04Z")