# Understanding tokenization for auto-complete **URL:** \ **Category:** Elasticsearch\ **Created:** [July 31, 2014, 9:20am UTC](https://discuss.elastic.co/t/understanding-tokenization-for-auto-complete/19002 "2014-07-31T09:20:42Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Petar\_Djekic](https://avatars.discourse-cdn.com/v4/letter/p/ecccb3/32.png) [@Petar\_Djekic](https://discuss.elastic.co/u/Petar_Djekic)\ **Post date:** [July 31, 2014, 9:20am UTC](https://discuss.elastic.co/t/understanding-tokenization-for-auto-complete/19002/1 "2014-07-31T09:20:42Z")
I'm using ES's auto-completion and i'd like to understand how the prefix tokenization works. Example queries and results its returning currently: 1. 'blackb' -\> 'Blackberry Q10 Red' --\> Expected 2. 'q' -\> 'Blackberry Q10 Red' --\> Expected 3. 'q10' -\> No result, expected is 'Blackberry Q10 Red' --\> Why are results returned when typing in 'q' but not 'q10'? 4. 'blackberry q10' --\> Expected 5. 'sam' -\> 'Samsung Galaxy S5' --\> Expected 6. 'galax' \> -\> 'Samsung Galaxy S5' --\> Expected 7. 'S5' -\> No result, expected is 'Samsung Galaxy S5' I'm indexing the document using input: ["blackberry, "Q10 Red"], input: ["samsung", "galaxy s5"], please find the mapping / query below. I thought the standard tokenizer would also tokenize on whitespaces and hence give result for S5, also i don't understand why 'q' gives results but 'q10' doesn't. Can i use the prefix tokenizer for such a use case or would it need to switch to ngrams completely? My mapping looks as follow: ``` "mappings" : { "suggestions" : { "_timestamp": { "enabled": true, "path" : "lastTimestamp" }, "properties" : { "suggest" : { "type" : "completion", "index_analyzer" : "standard", "search_analyzer" : "simple", "payloads" : true, "context" : { "type" : { "type" : "category", "path" : "entity" ``` and query like this: { ``` "suggestions" : { "text" : "'", "completion" : { "size" : 5, "field" : "suggest", "fuzzy" : { "fuzziness" : 1 }, "context" : { "type" : "'" } } } ``` } -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/4a917898-8525-4916-807e-cb72001b30c5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4a917898-8525-4916-807e-cb72001b30c5%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [August 4, 2014, 9:00am UTC](https://discuss.elastic.co/t/understanding-tokenization-for-auto-complete/19002/2 "2014-08-04T09:00:02Z")
Hey, two things: First, the completion suggester uses the simple analyzer by default. Please use the analyze API with some of your terms and you will see, why they dont match anymore (especially the q10). You may want to try out the inquisitor plugin for that, it has a nice web UI for the analyze API. Second: The completion suggester is a prefix suggester, so "galaxy S5' requires you to type "galaxy" first before typing "S5"... Hope this helps --Alex On Thu, Jul 31, 2014 at 11:20 AM, Petar Djekic \< [petar.djekic@rocket-internet.de](mailto:petar.djekic@rocket-internet.de)\> wrote: > I'm using ES's auto-completion and i'd like to understand how the prefix > tokenization works. Example queries and results its returning currently: > > 1. 'blackb' -\> 'Blackberry Q10 Red' > --\> Expected > > 2. 'q' -\> 'Blackberry Q10 Red' > --\> Expected > > 3. 'q10' -\> No result, expected is 'Blackberry Q10 Red' > --\> Why are results returned when typing in 'q' but not 'q10'? > > 4. 'blackberry q10' > --\> Expected > > 5. 'sam' -\> 'Samsung Galaxy S5' > --\> Expected > > 6. 'galax' \> -\> 'Samsung Galaxy S5' > --\> Expected > > 7. 'S5' > -\> No result, expected is 'Samsung Galaxy S5' > > I'm indexing the document using input: ["blackberry, "Q10 Red"], input: > ["samsung", "galaxy s5"], please find the mapping / query below. I thought > the standard tokenizer would also tokenize on whitespaces and hence give > result for S5, also i don't understand why 'q' gives results but 'q10' > doesn't. Can i use the prefix tokenizer for such a use case or would it > need to switch to ngrams completely? > > My mapping looks as follow: > > ``` > "mappings" : { > > "suggestions" : { > > "_timestamp": { > > "enabled": true, > > "path" : "lastTimestamp" > > }, > > "properties" : { > > "suggest" : { "type" : "completion", > > "index_analyzer" : "standard", > > "search_analyzer" : "simple", > > "payloads" : true, > > "context" : { > > "type" : { > > "type" : "category", > > "path" : "entity" > > ``` > > and query like this: > > { > > ``` > "suggestions" : { > > "text" : "'", > > "completion" : { > > "size" : 5, > > "field" : "suggest", > > "fuzzy" : { > > "fuzziness" : 1 > > }, > > "context" : { > > "type" : "'" > > } > > } > > } > > ``` > > } > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/4a917898-8525-4916-807e-cb72001b30c5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4a917898-8525-4916-807e-cb72001b30c5%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/4a917898-8525-4916-807e-cb72001b30c5%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/4a917898-8525-4916-807e-cb72001b30c5%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAGCwEM8X%2BjDEXx7KvvaQ-exhyasrn10JrLVn5aEaDWyAfsamKQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCwEM8X%2BjDEXx7KvvaQ-exhyasrn10JrLVn5aEaDWyAfsamKQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:11am UTC](https://discuss.elastic.co/t/understanding-tokenization-for-auto-complete/19002/3 "2017-07-06T01:11:08Z")