# Possible Issue with Kuromoji Tokenization when English/Japanese are present

**URL:** <https://discuss.elastic.co/t/possible-issue-with-kuromoji-tokenization-when-english-japanese-are-present/90680>\
**Category:** Elasticsearch\
**Created:** [June 23, 2017, 7:27pm UTC](https://discuss.elastic.co/t/possible-issue-with-kuromoji-tokenization-when-english-japanese-are-present/90680 "2017-06-23T19:27:20Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Bryan\_Warner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bryan_warner/32/1225_2.png) [@Bryan\_Warner](https://discuss.elastic.co/u/Bryan_Warner)\
**Post date:** [June 23, 2017, 7:27pm UTC](https://discuss.elastic.co/t/possible-issue-with-kuromoji-tokenization-when-english-japanese-are-present/90680/1 "2017-06-23T19:27:20Z")

</div>

Text =\> `Dior化粧品等の輸入総代理店で` , which is indexed with the default Kuromoji analyzer settings and produces the following tokens:

```auto
dior
start: 0 end: 4 pos: 0
化粧
start: 4 end: 6 pos: 1
品等
start: 6 end: 8 pos: 2
輸入
start: 9 end: 11 pos: 4
総
start: 11 end: 12 pos: 5
代理
start: 12 end: 14 pos: 6
店
start: 14 end: 15 pos: 7

```

However, we noticed that when a user searched for the term `Dior化粧品`, it did not produce a match (using same analyzer settings). The reason is that the search term is tokenized as such:

```auto
dior
start: 0 end: 4 pos: 0
化粧
start: 4 end: 6 pos: 1
品  
start: 6 end: 7 pos: 2

```

Since the word `cosmetics` is the Japanese term `化粧品`, it seems like the query got analyzed correctly but the piece of text produced an unexpected bigram sequence of `化粧` and `品等`

Not sure if this is a valid issue due to the mix of English/Japanese in the text or my Japanese fundamentals are off here

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 21, 2017, 7:27pm UTC](https://discuss.elastic.co/t/possible-issue-with-kuromoji-tokenization-when-english-japanese-are-present/90680/2 "2017-07-21T19:27:28Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.