# Stop standard tokenizer from splitting on punctuations

**URL:** <https://discuss.elastic.co/t/stop-standard-tokenizer-from-splitting-on-punctuations/300951>\
**Category:** Elasticsearch\
**Created:** [March 29, 2022, 11:28am UTC](https://discuss.elastic.co/t/stop-standard-tokenizer-from-splitting-on-punctuations/300951 "2022-03-29T11:28:00Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Nipun\_Tulsyan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nipun_tulsyan/32/99437_2.png) [@Nipun\_Tulsyan](https://discuss.elastic.co/u/Nipun_Tulsyan)\
**Post date:** [March 29, 2022, 11:28am UTC](https://discuss.elastic.co/t/stop-standard-tokenizer-from-splitting-on-punctuations/300951/1 "2022-03-29T11:28:00Z")

</div>

Hi

We use the "standard" tokenizer in custom analyzer definitions. By default  
the standard tokenizer splits words on hyphens and ampersands, so for  
example "i-mac" is tokenized to "i" and "mac"

Is there any way to configure the behaviour of the standard tokenizer to  
stop it splitting words on all punctuations except comma(","), while still doing all  
the other tokenizing it does.

Or maybe define a custom tokenizer which will achieve the above  
For example:  
query string ["n-12"] should create a this token:

```auto
{
  "tokens" : [
    {
      "token" : "n-12",
      "start_offset" : 0,
      "end_offset" : 2,
      "type" : "<ALPHANUM>",
      "position" : 0
    }
  ]
} 
instead of 
{
    "tokens" : [
      {
        "token" : "n",
        "start_offset" : 0,
        "end_offset" : 1,
        "type" : "<ALPHANUM>",
        "position" : 0
      },
      {
        "token" : "12",
        "start_offset" : 0,
        "end_offset" : 2,
        "type" : "<ALPHANUM>",
        "position" : 0
      }
    ]
}

```

Regards,  
Nipun

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 26, 2022, 11:28am UTC](https://discuss.elastic.co/t/stop-standard-tokenizer-from-splitting-on-punctuations/300951/2 "2022-04-26T11:28:17Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.