# One token multiple normalizations

**URL:** <https://discuss.elastic.co/t/one-token-multiple-normalizations/51312>\
**Category:** Elasticsearch\
**Created:** [May 30, 2016, 11:26am UTC](https://discuss.elastic.co/t/one-token-multiple-normalizations/51312 "2016-05-30T11:26:53Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![drount](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/drount/32/10022_2.png) [@drount](https://discuss.elastic.co/u/drount)\
**Post date:** [May 30, 2016, 11:26am UTC](https://discuss.elastic.co/t/one-token-multiple-normalizations/51312/1 "2016-05-30T11:26:53Z")

</div>

Hello,

Im setting up a project where I need to have several normalizations for a given token.

For example, the text:

I Don't go

should be result of the queries:

I dont go  
I don t go  
I don't go

Thus I need the token "Don't" to be normalized to "dont" and "don t".

Stemming is not an option as there are tokens that are invented words (ex. C.O.R.E, should be normalized to CORE and C O R E)

Any idea on how to solve this?

Thank you!

---

<div class="post-metadata">

**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\
**Post date:** [May 30, 2016, 12:56pm UTC](https://discuss.elastic.co/t/one-token-multiple-normalizations/51312/2 "2016-05-30T12:56:56Z")

</div>

This probably requires a custom token filter that inserts synonyms.

---

<div class="post-metadata">

**Author:** ![drount](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/drount/32/10022_2.png) [@drount](https://discuss.elastic.co/u/drount)\
**Post date:** [May 30, 2016, 1:10pm UTC](https://discuss.elastic.co/t/one-token-multiple-normalizations/51312/3 "2016-05-30T13:10:00Z")

</div>

I'm quite surprised I haven't found anything similar. To me, this looks like a problem that lots of system can potentially have.

---

<div class="post-metadata">

**Author:** ![pratyusha](https://avatars.discourse-cdn.com/v4/letter/p/9dc877/32.png) [@pratyusha](https://discuss.elastic.co/u/pratyusha)\
**Post date:** [May 30, 2016, 1:22pm UTC](https://discuss.elastic.co/t/one-token-multiple-normalizations/51312/4 "2016-05-30T13:22:49Z")

</div>

You need to use a custom filter to replace the character ' with what ever you want.

[https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-pattern-replace-charfilter.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-pattern-replace-charfilter.html)

Eg:

```
    "char_filter": {
      "my_pattern": {
        "type": "pattern_replace",
        "pattern": "([-()',])",
        "replacement": ""
      },
      "my_mapping": {
        "type": "mapping",
        "mappings_path": "/etc/elasticsearch/replace_words.txt"
      }
    }
```

---

<div class="post-metadata">

**Author:** ![drount](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/drount/32/10022_2.png) [@drount](https://discuss.elastic.co/u/drount)\
**Post date:** [May 30, 2016, 1:30pm UTC](https://discuss.elastic.co/t/one-token-multiple-normalizations/51312/5 "2016-05-30T13:30:53Z")

</div>

The problem with that solution is that I lose the original one.

As an example:

with "Don't" I can pattern\_replace ' with (empty) to get "Dont"

However, the token "Don't" is now lost and thus, I cannot not get "Don t".

In addition, I want this to be Token based and char filtering is pre-tokenizing.

---

<div class="post-metadata">

**Author:** ![pratyusha](https://avatars.discourse-cdn.com/v4/letter/p/9dc877/32.png) [@pratyusha](https://discuss.elastic.co/u/pratyusha)\
**Post date:** [May 30, 2016, 1:58pm UTC](https://discuss.elastic.co/t/one-token-multiple-normalizations/51312/6 "2016-05-30T13:58:13Z")

</div>

What if you use the same analyser for your search query also? Original data anyway will be there in source.

For "I don t go" also if you want "I don't go" as the result , then ngram analyser also shall be used I guess.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 10:47pm UTC](https://discuss.elastic.co/t/one-token-multiple-normalizations/51312/7 "2017-07-05T22:47:59Z")

</div>