# How to whitelist terms in a custom analyzer

**URL:** <https://discuss.elastic.co/t/how-to-whitelist-terms-in-a-custom-analyzer/92734>\
**Category:** Elasticsearch\
**Created:** [July 11, 2017, 11:35pm UTC](https://discuss.elastic.co/t/how-to-whitelist-terms-in-a-custom-analyzer/92734 "2017-07-11T23:35:53Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![rlvoyer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rlvoyer/32/4928_2.png) [@rlvoyer](https://discuss.elastic.co/u/rlvoyer)\
**Post date:** [July 11, 2017, 11:35pm UTC](https://discuss.elastic.co/t/how-to-whitelist-terms-in-a-custom-analyzer/92734/1 "2017-07-11T23:35:53Z")

</div>

Hi there,

I'm using the edge ngram tokenizer to support search autocomplete. Unlike other tokenizers, the `edge_ngram` tokenizer is relatively limited in terms of its configurability. Take for a example the string "AT&T". Even with "symbols" included in an edge ngram tokenizer's `token_chars` field, that string would be tokenized as follows:

```
 {
  "tokens": [
    {
      "token": "a",
      "start_offset": 0,
      "end_offset": 1,
      "type": "word",
      "position": 0
    },
    {
      "token": "at",
      "start_offset": 0,
      "end_offset": 2,
      "type": "word",
      "position": 1
    },
    {
      "token": "t",
      "start_offset": 3,
      "end_offset": 4,
      "type": "word",
      "position": 2
    }
  ]
}

```

Tokenizing the ampersand away results in 3 very high frequency n-grams (2 unigrams, 1 bigram). The problem with this then is that any search for this string is likely to return a lot of false positives. It seems to me that there should be a token whitelist filter that allows you to preserve select terms/phrases (ie. exceptions to tokenization). I've noticed that there is a filter that will prevent stemming -- the keyword marker token filter -- but I've found no equivalent for allowing exceptions to tokenization rules.

Any tips would be greatly appreciated, thanks!

---

<div class="post-metadata">

**Author:** ![polyfractal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/polyfractal/32/48162_2.png) [@polyfractal](https://discuss.elastic.co/u/polyfractal)\
**Post date:** [July 12, 2017, 4:54pm UTC](https://discuss.elastic.co/t/how-to-whitelist-terms-in-a-custom-analyzer/92734/2 "2017-07-12T16:54:52Z")

</div>

Yeah, the `ngram` tokenizer can be somewhat limiting, but mainly because it's designed as a quick/dirty ngrammer when you just want simple behavior.

For your case, you'd probably be better served by using a less aggressive tokenizer (something like `whitespace`) and then an `ngram` token filter. The `whitespace` tokenizer will preserve the ampersand, and then the `ngram` filter will do as you expect.

---

<div class="post-metadata">

**Author:** ![rlvoyer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rlvoyer/32/4928_2.png) [@rlvoyer](https://discuss.elastic.co/u/rlvoyer)\
**Post date:** [July 12, 2017, 5:03pm UTC](https://discuss.elastic.co/t/how-to-whitelist-terms-in-a-custom-analyzer/92734/3 "2017-07-12T17:03:49Z")

</div>

I considered this, but it doesn't feel like the right approach because these sorts of strings are very much edge cases. In general, tokenizing on ampersands feels like the right thing to do.

---

<div class="post-metadata">

**Author:** ![rlvoyer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rlvoyer/32/4928_2.png) [@rlvoyer](https://discuss.elastic.co/u/rlvoyer)\
**Post date:** [July 13, 2017, 8:24pm UTC](https://discuss.elastic.co/t/how-to-whitelist-terms-in-a-custom-analyzer/92734/4 "2017-07-13T20:24:11Z")

</div>

That said, I wasn't aware of the existence of the edge ngram token filter (as an alternative to the edge ngram tokenizer), so thanks for the heads up.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 10, 2017, 8:24pm UTC](https://discuss.elastic.co/t/how-to-whitelist-terms-in-a-custom-analyzer/92734/5 "2017-08-10T20:24:29Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.