# Configuring the standard tokenizer **URL:** \ **Category:** Elasticsearch\ **Created:** [August 9, 2012, 11:09am UTC](https://discuss.elastic.co/t/configuring-the-standard-tokenizer/8691 "2012-08-09T11:09:12Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Robin\_Hughes](https://avatars.discourse-cdn.com/v4/letter/r/e36b37/32.png) [@Robin\_Hughes](https://discuss.elastic.co/u/Robin_Hughes)\ **Post date:** [August 9, 2012, 11:09am UTC](https://discuss.elastic.co/t/configuring-the-standard-tokenizer/8691/1 "2012-08-09T11:09:12Z")
Hi We use the "standard" tokenizer in custom analyzer definitions. By default the standard tokenizer splits words on hyphens and ampersands, so for example "i-mac" is tokenized to "i" and "mac" Is there any way to configure the behaviour of the standard tokenizer to stop it splitting words on hyphens and ampersands, while still doing all the normal tokenizing it does on other punctuation? We could define our own "Pattern" tokenizer using a regex but it seems like any custom regex we write is unlikely to deal with the myriad unicode characters the standard tokenizer handles already. Regards, Robin ---
**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\ **Post date:** [August 9, 2012, 5:23pm UTC](https://discuss.elastic.co/t/configuring-the-standard-tokenizer/8691/2 "2012-08-09T17:23:12Z")
The standard tokenizer is following the Unicode Standard Annex #29, and doesn't really have any settings besides version and max\_token\_length. I am not sure what's your use case, but one possible solution that comes to mind would be to replace hyphens and ampersands with symbols that don't cause words to be split, or just remove them. For example, you can replace hyphens with underscores and remove ampersands. Another interesting replacement character is ".". You can achieve it using Mapping Char Filter[http://www.elasticsearch.org/guide/reference/index-modules/analysis/mapping-charfilter.html](http://www.elasticsearch.org/guide/reference/index-modules/analysis/mapping-charfilter.html) . $ curl -XPUT [http://localhost:9200/test](http://localhost:9200/test) -d '{ "settings" : { "index" : { "number\_of\_shards" : 5, "number\_of\_replicas" : 0, "analysis" : { "char\_filter" : { "my\_filter" : { "type" : "mapping", "mappings" : ["'''=\>", "-=\>\_"] } }, "analyzer" : { "my\_standard" : { "type": "custom", "tokenizer" : "standard", "char\_filter" : ["my\_filter"], "filter" : ["standard", "lowercase", "stop"] } } } } } }' {"ok":true,"acknowledged":true} $ curl "localhost:9200/test/\_analyze?analyzer=my\_standard&pretty=true" -d "Robin's i-mac is nice." { "tokens" : [ { "token" : "robins", "start\_offset" : 0, "end\_offset" : 7, "type" : "", "position" : 1 }, { "token" : "i\_mac", "start\_offset" : 8, "end\_offset" : 13, "type" : "", "position" : 2 }, { "token" : "nice", "start\_offset" : 17, "end\_offset" : 21, "type" : "", "position" : 4 } ] } On Thursday, August 9, 2012 7:09:12 AM UTC-4, Robin Hughes wrote: > Hi > > We use the "standard" tokenizer in custom analyzer definitions. By default > the standard tokenizer splits words on hyphens and ampersands, so for > example "i-mac" is tokenized to "i" and "mac" > > Is there any way to configure the behaviour of the standard tokenizer to > stop it splitting words on hyphens and ampersands, while still doing all > the normal tokenizing it does on other punctuation? > > We could define our own "Pattern" tokenizer using a regex but it seems > like any custom regex we write is unlikely to deal with the myriad unicode > characters the standard tokenizer handles already. > > Regards, > Robin ---
**Author:** ![Robin\_Hughes](https://avatars.discourse-cdn.com/v4/letter/r/e36b37/32.png) [@Robin\_Hughes](https://discuss.elastic.co/u/Robin_Hughes)\ **Post date:** [August 10, 2012, 2:07pm UTC](https://discuss.elastic.co/t/configuring-the-standard-tokenizer/8691/3 "2012-08-10T14:07:23Z")
Thanks Igor for a very elegant solution! Regards, Robin ---
**Author:** ![Roeland\_Van\_Heddegem](https://avatars.discourse-cdn.com/v4/letter/r/82dd89/32.png) [@Roeland\_Van\_Heddegem](https://discuss.elastic.co/u/Roeland_Van_Heddegem)\ **Post date:** [June 19, 2015, 12:54pm UTC](https://discuss.elastic.co/t/configuring-the-standard-tokenizer/8691/4 "2015-06-19T12:54:27Z")
Hi, What about a simple query string query like this: ``` { "_source":true, "query":{ "simple_query_string":{ "query":"i-m*", "default_operator":"AND" } } } ``` This should match "Robin's i-mac is nice.", but I get no results. Why? Roeland ---
**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\ **Post date:** [June 20, 2015, 9:25pm UTC](https://discuss.elastic.co/t/configuring-the-standard-tokenizer/8691/5 "2015-06-20T21:25:54Z")
By default the `simple_query_string` query doesn't analyze the words with wildcards. As a result it searches for all tokens that start with `i-ma`. The word `i-mac` doesn't match this request because during analysis it's split into two tokens `i` and `mac` and neither of these tokens starts with `i-ma`. In order to make this query find `i-mac` you need to make it analyze wildcards: ``` { "_source":true, "query":{ "simple_query_string":{ "query":"i-ma*", "analyze_wildcard": true, "default_operator":"AND" } } } ``` Igor ---
**Author:** ![Roeland\_Van\_Heddegem](https://avatars.discourse-cdn.com/v4/letter/r/82dd89/32.png) [@Roeland\_Van\_Heddegem](https://discuss.elastic.co/u/Roeland_Van_Heddegem)\ **Post date:** [June 22, 2015, 8:23am UTC](https://discuss.elastic.co/t/configuring-the-standard-tokenizer/8691/6 "2015-06-22T08:23:39Z")
Thanks for answering Igor, it works. But, I couldn't find this parameter in the documentation of simple\_query\_string query? see [Simple Query String Query](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-simple-query-string-query.html) PS: I had also made a question on stackoverflow: [ElasticSearch - Searching with hyphens](http://stackoverflow.com/questions/30917043/elasticsearch-searching-with-hyphens/30919206#30919206), that I answered myself by quoting you. ---
**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\ **Post date:** [June 22, 2015, 10:51pm UTC](https://discuss.elastic.co/t/configuring-the-standard-tokenizer/8691/7 "2015-06-22T22:51:09Z")
> [@Roeland\_Van\_Heddegem](#): > > But, I couldn't find this parameter in the documentation of simple\_query\_string query? Indeed, this parameter was added in 1.5, but documentation wasn't updated back then. I fixed that. ---
**Author:** ![prayas](https://avatars.discourse-cdn.com/v4/letter/p/a8b319/32.png) [@prayas](https://discuss.elastic.co/u/prayas)\ **Post date:** [August 11, 2016, 11:29am UTC](https://discuss.elastic.co/t/configuring-the-standard-tokenizer/8691/8 "2016-08-11T11:29:58Z")
Hi I have to search order number which is like XXXX-XXX. - is separating this into separate tokens. I used charfilter to make it as token XXXX\_XXX so that i could search that order number as something like XXXX\_X\*. This is what i had done. Am i missing something?? I am not getting the desired result. I am using version 1.4.2. { "settings": { "analysis": { "char\_filter": { "myfilter": { "type": "mapping", "mappings": ["-=\>\_"] }}, ``` "analyzer": { "special_analyzer": { "type": "custom", "tokenizer" : "standard", "char_filter": ["myfilter"], "filter": ["standard", "lowercase", "stop"] } } } ``` }, "type" : "jdbc", "jdbc" : { "strategy" : "simple", "url" : "jdbc:datadirect:openedge://10.0.1.169:2032;databaseName=admin11.db", "user" : "sysprogress", "password" : "sysprogress", "schedule" : "0 0-59 0-23 ? \* \*", "sql" : "SELECT 'order,' + cast(order.rowid as varchar(255)) as '\_id' ,order.ordnr,order.sltl from PUB.order", "index" : "admin11", "type" : "order", "order":{ "properties":{ "ordnr":{ ``` "type":"string", "analyzer":"special_analyzer" } } } } ``` } ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:28pm UTC](https://discuss.elastic.co/t/configuring-the-standard-tokenizer/8691/9 "2017-07-05T22:28:26Z")