# Tokenizer in Elasticsearch

**URL:** <https://discuss.elastic.co/t/tokenizer-in-elasticsearch/75933>\
**Category:** Elasticsearch\
**Created:** [February 21, 2017, 5:38pm UTC](https://discuss.elastic.co/t/tokenizer-in-elasticsearch/75933 "2017-02-21T17:38:41Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![ashit\_pupu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashit_pupu/32/101113_2.png) [@ashit\_pupu](https://discuss.elastic.co/u/ashit_pupu)\
**Post date:** [February 21, 2017, 5:38pm UTC](https://discuss.elastic.co/t/tokenizer-in-elasticsearch/75933/1 "2017-02-21T17:38:41Z")

</div>

The issue is I have to tokenize data into tokens based on spaces at the same time I can't tokenize the data based on special characters. Right now the regex I have is

```
   (\w*[-*#+=;:\/,~_]*\w+)

```

With this when I process the string

```
1-CHECK ON BLOCKS BELOW IF MARKET CORRECTION ARE LOADED: PCORP:BLOCK=ANCTRLG&amp;V5PTCLG; AF55722 BRTBMWA-3289 (AF55722) in block ANCTRLG (Product ID: CAAZ 107 4493 R1A10 ) AF55736 BRTBMWA-3290 (AF55726)in block V5PTCLG (Product ID: CAAZ 107 4260 R2A08 ) IF MARKET CORRECTIONS ARE LOADED THEN V5 INTERFACE PROPERTY MUST BE DEFINED AS FOLLOW : MUXFIM : ACC-OFF (Accelerate Alligment is not active) WLL : ACC-ON (Accelerate Alligment is active ) : EXAPC:V5ID=v5id,PROP=ACC-OFF; 

```

What it does is tokenizes the string based on spaces at the same time it also tokenizes the data based on special character like

```
         : EXAPC:V5ID=v5id is tokenized to : EXAPC, :V5ID and =v5id rather want it to split as : and EXAPC:V5ID=v5id

```

I want to avoid this any idea on this any help will be appreciated.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 21, 2017, 5:38pm UTC](https://discuss.elastic.co/t/tokenizer-in-elasticsearch/75933/2 "2017-03-21T17:38:41Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.