# Tokenization help mixed n-grams **URL:** \ **Category:** Elasticsearch\ **Created:** [February 26, 2015, 12:46pm UTC](https://discuss.elastic.co/t/tokenization-help-mixed-n-grams/22407 "2015-02-26T12:46:37Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Ilija\_Subasic](https://avatars.discourse-cdn.com/v4/letter/i/ee7513/32.png) [@Ilija\_Subasic](https://discuss.elastic.co/u/Ilija_Subasic)\ **Post date:** [February 26, 2015, 12:46pm UTC](https://discuss.elastic.co/t/tokenization-help-mixed-n-grams/22407/1 "2015-02-26T12:46:37Z")
Hi, I am trying to create a tokenizer that is going to create a tokens looking something like this: "ab c dd c" would be tokenized as "ab", "abc", "abcd", "abcdd", "abcddc", "cd", "cdd", "cddc", "dd", "ddc" so basically I need something that is going to do an ngram indexing from the start of each token. This is different then edge n-gram which will tokenize each token separatelly. Any ideas on how to do this without coding a specific tokenizer. Thanks, Ilija -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/ec72da21-ed99-4cc8-829c-058467c020a5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ec72da21-ed99-4cc8-829c-058467c020a5%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![masaru](https://avatars.discourse-cdn.com/v4/letter/m/a9a28c/32.png) [@masaru](https://discuss.elastic.co/u/masaru)\ **Post date:** [February 27, 2015, 7:55am UTC](https://discuss.elastic.co/t/tokenization-help-mixed-n-grams/22407/2 "2015-02-27T07:55:40Z")
Hi, You can use mapping char filter to remove white space and then ngram tokenises with min\_gram=2/max\_gram= to make it ngrams. (not sure if you’d like to omit “bc”, “bcd”… or not though) Masaru On February 26, 2015 at 21:46:42, Ilija Subasic ([subasic.ilija@gmail.com](mailto:subasic.ilija@gmail.com)) wrote: > Hi, > I am trying to create a tokenizer that is going to create a tokens > looking something like this: > > "ab c dd c" would be tokenized as "ab", "abc", "abcd", "abcdd", "abcddc", > "cd", "cdd", "cddc", "dd", "ddc" > > so basically I need something that is going to do an ngram indexing from > the start of each token. This is different > then edge n-gram which will tokenize each token separatelly. > Any ideas on how to do this without coding a specific tokenizer. > > Thanks, > Ilija > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" > group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/ec72da21-ed99-4cc8-829c-058467c020a5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ec72da21-ed99-4cc8-829c-058467c020a5%40googlegroups.com). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/etPan.54f02339.625558ec.129%40citra.local](https://groups.google.com/d/msgid/elasticsearch/etPan.54f02339.625558ec.129%40citra.local). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:29am UTC](https://discuss.elastic.co/t/tokenization-help-mixed-n-grams/22407/3 "2017-07-06T00:29:36Z")