# Elastic Edge\_Ngram with Whitespace Word Breaker

**URL:** <https://discuss.elastic.co/t/elastic-edge-ngram-with-whitespace-word-breaker/225280>\
**Category:** Elasticsearch\
**Created:** [March 26, 2020, 8:17pm UTC](https://discuss.elastic.co/t/elastic-edge-ngram-with-whitespace-word-breaker/225280 "2020-03-26T20:17:23Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![cawoodm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cawoodm/32/14083_2.png) [@cawoodm](https://discuss.elastic.co/u/cawoodm)\
**Post date:** [March 26, 2020, 8:17pm UTC](https://discuss.elastic.co/t/elastic-edge-ngram-with-whitespace-word-breaker/225280/1 "2020-03-26T20:17:23Z")

</div>

We want our edge\_ngram search to only to word break on whitespace.

If we use:

```auto
    "tokenizer": {
        "token_chars": ["letter", "digit", "punctuation", "symbol"],
        "min_gram": "2",
        "type": "edge_ngram",
        "max_gram": "20"
    }

```

Then it breaks words on things not considered\* symbols (like `=`).

How can we use the edge\_ngram tokenizer only tokenizing (word breaking) on whitespace.

Note: `"token_chars": ["whitespace"],` produces no tokens at all.

Test case:  
`"some foo/text with=this.kind?of_thing"` should produce `["some", "foo/text", "with=this.kind?of_thing"]`

Bonus Question: Which characters _are_ considered symbols. It's not obvious from the source code.

---

<div class="post-metadata">

**Author:** ![cawoodm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cawoodm/32/14083_2.png) [@cawoodm](https://discuss.elastic.co/u/cawoodm)\
**Post date:** [March 27, 2020, 9:05am UTC](https://discuss.elastic.co/t/elastic-edge-ngram-with-whitespace-word-breaker/225280/2 "2020-03-27T09:05:30Z")

</div>

OK, found a solution using the rather poorly documented `custom_token_chars`:

```auto
"tokenizer": {
        "token_chars": ["letter", "digit", "punctuation", "symbol"],
        "custom_token_chars": ["="],
        "min_gram": "2",
        "type": "edge_ngram",
        "max_gram": "20"
    }

```

I can confirm that at least the following character are already covered by `punctutation` and `symbols` above:  
`/ * - + „ “ * % & ! . , ; : ( ) ° ß ø`

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [March 30, 2020, 2:52pm UTC](https://discuss.elastic.co/t/elastic-edge-ngram-with-whitespace-word-breaker/225280/3 "2020-03-30T14:52:34Z")

</div>

dumb question: If you only want to tokenize on whitespace, why not use a [whitespace tokenizer](https://www.elastic.co/guide/en/elasticsearch/reference/7.6/analysis-whitespace-tokenizer.html)? I guess there is some more logic done on your side?

---

<div class="post-metadata">

**Author:** ![cawoodm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cawoodm/32/14083_2.png) [@cawoodm](https://discuss.elastic.co/u/cawoodm)\
**Post date:** [March 31, 2020, 10:14am UTC](https://discuss.elastic.co/t/elastic-edge-ngram-with-whitespace-word-breaker/225280/4 "2020-03-31T10:14:51Z")

</div>

Indeed, I was unaware how to combine filters, analyzers and tokenizers but I got some feedback on GitHub which resulted in these settings:

```auto
    "settings": {
        "index": {
            "analysis": {
                "analyzer": {
                    "whitesp": {
                        "tokenizer": "whitespace",
                        "filter": [
                            "lowercase"
                        ]
                    },
                    "edgegram": {
                    	"tokenizer": "whitespace",
                        "filter": [
                            "lowercase",
                            "edgegram"
                        ]
                    }
                },
                "filter": {
                    "edgegram": {
                        "min_gram": "2",
                        "type": "edge_ngram",
                        "max_gram": "20"
                    }
                }
            }
        }
    },
    "mappings": {
        "dynamic": "strict",
        "properties": {
            "text_en": {
                "search_analyzer": "whitesp",
                "analyzer": "edgegram",
                "type": "text"
            },
            "text_de": {
                "search_analyzer": "whitesp",
                "analyzer": "edgegram",
                "type": "text"
            }
            ...

```

As I understand it this ensures the analysis/indexer uses edge ngram tokenizing on whitespace whilst the search analyser also tokenizes on whitespace but does not to the ngram stuff.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 28, 2020, 10:14am UTC](https://discuss.elastic.co/t/elastic-edge-ngram-with-whitespace-word-breaker/225280/5 "2020-04-28T10:14:52Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.