# Prevent standard tokenizer from tokenizing \<IDEOGRAPHIC\> token per character

**URL:** <https://discuss.elastic.co/t/prevent-standard-tokenizer-from-tokenizing-ideographic-token-per-character/377797>\
**Category:** Elasticsearch\
**Created:** [May 5, 2025, 6:55am UTC](https://discuss.elastic.co/t/prevent-standard-tokenizer-from-tokenizing-ideographic-token-per-character/377797 "2025-05-05T06:55:42Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![yukha-dw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yukha-dw/32/142929_2.png) [@yukha-dw](https://discuss.elastic.co/u/yukha-dw)\
**Post date:** [May 5, 2025, 6:55am UTC](https://discuss.elastic.co/t/prevent-standard-tokenizer-from-tokenizing-ideographic-token-per-character/377797/1 "2025-05-05T06:55:42Z")

</div>

Hello, is it possible to use N-Gram Filter on each **\<IDEOGRAPHIC\>** phrase just like how it works on **\<HANGUL\>**?

For example (min\_gram=max\_gram=2 & preserve\_original):  
Input: "我 爱 青苹果"  
Desired Output: "我", "爱", "青苹", "苹果", and "青苹果"

1. If I set ngram as tokenizer, it will include whitespace too:  
Setting:

```auto
{
          "type": "custom",
          "tokenizer": "ngram_tokenizer_2_2"
}

```

Output: **"我\<WHITESPACE\>" (undesired), "\<WHITESPACE\>爱" (undesired), "爱\<WHITESPACE\>" (undesired),"\<WHITESPACE\>青" (undesired)**,"青苹", and "苹果"

1. If I set ngram as token filter and use standard tokenizer, it will split "青苹果" per character instead:  
Setting:

```auto
{
          "type": "custom",
          "tokenizer": "standard"
          "filter": ["ngram_filter_2_2"]
}

```

Output: "我", "爱", **"青" (undesired)**, **"苹" (undesired)**, and **"果" (undesired)**

1. If I set ngram as token filter and use whitespace tokenizer, it will return tokens as expected, but it will fail on multilanguage input such as "I love青苹果"  
Setting:

```auto
{
          "type": "custom",
          "tokenizer": "whitespace"
          "filter": ["ngram_filter_2_2"]
}

```

Output: "I", "lo", "ov", "ve", **"e青" (undesired)**, "青苹", "苹果", and **"love青苹果" (undesired)**

Thank you in advance!

---

<div class="post-metadata">

**Author:** ![yukha-dw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yukha-dw/32/142929_2.png) [@yukha-dw](https://discuss.elastic.co/u/yukha-dw)\
**Post date:** [May 8, 2025, 9:56am UTC](https://discuss.elastic.co/t/prevent-standard-tokenizer-from-tokenizing-ideographic-token-per-character/377797/2 "2025-05-08T09:56:32Z")

</div>

up up!