# Standard tokenizer documentation doesn't match behavior

**URL:** <https://discuss.elastic.co/t/standard-tokenizer-documentation-doesnt-match-behavior/6440>\
**Category:** Elasticsearch\
**Created:** [January 19, 2012, 1:05pm UTC](https://discuss.elastic.co/t/standard-tokenizer-documentation-doesnt-match-behavior/6440 "2012-01-19T13:05:44Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![George\_Sakkis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/george_sakkis/32/2801_2.png) [@George\_Sakkis](https://discuss.elastic.co/u/George_Sakkis)\
**Post date:** [January 19, 2012, 1:05pm UTC](https://discuss.elastic.co/t/standard-tokenizer-documentation-doesnt-match-behavior/6440/1 "2012-01-19T13:05:44Z")

</div>

Am I reading it wrong or has something changed recently? From  
[http://www.elasticsearch.org/guide/reference/index-modules/analysis/standard-tokenizer.html:](http://www.elasticsearch.org/guide/reference/index-modules/analysis/standard-tokenizer.html:)

"[A standard tokenizer] also splits words at hyphens, unless there’s a  
number in the token, in which case the whole token is interpreted as a  
product number and is not split."

curl -XGET '[http://localhost:9200/twitter/\_analyze/](http://localhost:9200/twitter/_analyze/)?  
pretty=true&analyzer=standard' -d '123-456-7890'  
{  
"tokens" : [ {  
"token" : "123",  
"start\_offset" : 0,  
"end\_offset" : 3,  
"type" : "",  
"position" : 1  
}, {  
"token" : "456",  
"start\_offset" : 4,  
"end\_offset" : 7,  
"type" : "",  
"position" : 2  
}, {  
"token" : "7890",  
"start\_offset" : 8,  
"end\_offset" : 12,  
"type" : "",  
"position" : 3  
} ]

"It recognizes email addresses and internet hostnames as one token."

curl -GET '[http://localhost:9200/twitter/\_analyze/](http://localhost:9200/twitter/_analyze/)?  
pretty=true&analyzer=standard' -d 'somebody@example.com'  
{  
"tokens" : [ {  
"token" : "somebody",  
"start\_offset" : 0,  
"end\_offset" : 8,  
"type" : "",  
"position" : 1  
}, {  
"token" : "[example.com](http://example.com)",  
"start\_offset" : 9,  
"end\_offset" : 20,  
"type" : "",  
"position" : 2  
} ]

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [January 19, 2012, 6:36pm UTC](https://discuss.elastic.co/t/standard-tokenizer-documentation-doesnt-match-behavior/6440/2 "2012-01-19T18:36:30Z")

</div>

You are right, this is the old behavior of the tokenizer (though you can  
retain the email behavior by using the uax tokenizer). I will fix it  
shortly.

On Thu, Jan 19, 2012 at 3:05 PM, George Sakkis [george.sakkis@gmail.com](mailto:george.sakkis@gmail.com)wrote:

> Am I reading it wrong or has something changed recently? From
> 
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/analysis/standard-tokenizer.html)  
> :
> 
> "[A standard tokenizer] also splits words at hyphens, unless there’s a  
> number in the token, in which case the whole token is interpreted as a  
> product number and is not split."
> 
> curl -XGET '[http://localhost:9200/twitter/\_analyze/](http://localhost:9200/twitter/_analyze/)?  
> pretty=true&analyzer=standard' -d '123-456-7890'  
> {  
> "tokens" : [ {  
> "token" : "123",  
> "start\_offset" : 0,  
> "end\_offset" : 3,  
> "type" : "",  
> "position" : 1  
> }, {  
> "token" : "456",  
> "start\_offset" : 4,  
> "end\_offset" : 7,  
> "type" : "",  
> "position" : 2  
> }, {  
> "token" : "7890",  
> "start\_offset" : 8,  
> "end\_offset" : 12,  
> "type" : "",  
> "position" : 3  
> } ]
> 
> "It recognizes email addresses and internet hostnames as one token."
> 
> curl -GET '[http://localhost:9200/twitter/\_analyze/](http://localhost:9200/twitter/_analyze/)?  
> pretty=true&analyzer=standard' -d 'somebody@example.com'  
> {  
> "tokens" : [ {  
> "token" : "somebody",  
> "start\_offset" : 0,  
> "end\_offset" : 8,  
> "type" : "",  
> "position" : 1  
> }, {  
> "token" : "[example.com](http://example.com)",  
> "start\_offset" : 9,  
> "end\_offset" : 20,  
> "type" : "",  
> "position" : 2  
> } ]

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 3:42am UTC](https://discuss.elastic.co/t/standard-tokenizer-documentation-doesnt-match-behavior/6440/3 "2017-07-06T03:42:05Z")

</div>