# Whitespace tokenizer

**URL:** <https://discuss.elastic.co/t/whitespace-tokenizer/240165>\
**Category:** Elasticsearch\
**Created:** [July 7, 2020, 1:07pm UTC](https://discuss.elastic.co/t/whitespace-tokenizer/240165 "2020-07-07T13:07:27Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![anthony\_bliecq](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anthony_bliecq/32/48937_2.png) [@anthony\_bliecq](https://discuss.elastic.co/u/anthony_bliecq)\
**Post date:** [July 7, 2020, 1:07pm UTC](https://discuss.elastic.co/t/whitespace-tokenizer/240165/1 "2020-07-07T13:07:27Z")

</div>

Hi,

I'm actually using standard tokenizer wich contains whitespace tokenizer.  
The problem is that i want to separate a sentence by tokens with whitespace but i also want the entire sentence as token. How is this possible ?

# Concrete example:

want to parse "Hello How Are You" in these tokens --\> ["hello","how","are","you","hello how are you"].

# My actual config :

```auto
{
    "analysis": {
      "analyzer": {
        "smd_analyzer": {
          "type": "custom", 
          "tokenizer": "standard",
          "char_filter": [
            "html_strip",
            "smd_filter",
          ],
          "filter": [
            "lowercase",
            "asciifolding",
            "smd_length",
            "smd_stop"
          ]
        }
      },
      "char_filter": {
	        "smd_filter": {
	          "type": "pattern_replace",
	          "pattern": "(\\p{L}+)'(\\p{L}+)",
	          "replacement": "$0 $1 $2"
	        }
      },
      	"filter":{
	        "smd_length": {
	          "type": "length",
	          "min": 2
	        },
	  	    "smd_stop": {
		      "type": "stop",
	          "ignore_case": true,
	          "stopwords": ["LE", "LA", "LES", "DU", "DES", "OU", "ET", "SI", "STE", "CIE","SOC", "GEN", "GIE", "NV", "SA", "SARL", "ST", "BS", "CP", "CV","DA", "DS", "OAT", "TP", "TSDI", "TSR", "ZZ"]
		    }
      	}
    }
}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 4, 2020, 1:07pm UTC](https://discuss.elastic.co/t/whitespace-tokenizer/240165/2 "2020-08-04T13:07:28Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.