# Asian characters and not words are tokenized - CJK Analysis and Tokenization Problems **URL:** \ **Category:** Elasticsearch\ **Created:** [March 10, 2011, 4:00am UTC](https://discuss.elastic.co/t/asian-characters-and-not-words-are-tokenized-cjk-analysis-and-tokenization-problems/4072 "2011-03-10T04:00:18Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Wolf\_2](https://avatars.discourse-cdn.com/v4/letter/w/ac8455/32.png) [@Wolf\_2](https://discuss.elastic.co/u/Wolf_2)\ **Post date:** [March 10, 2011, 4:00am UTC](https://discuss.elastic.co/t/asian-characters-and-not-words-are-tokenized-cjk-analysis-and-tokenization-problems/4072/1 "2011-03-10T04:00:18Z")
I am using the CJK Analyzer to index Japanese language content into elasticsearch: However the the input is not tokenized into words but rather characters. The CJK analyzer does not have this problem when using a stand alone lucene demonstration program, pp. 147, "Lucene in Action" second edition, Manning 2010. I also do not have any problems with Sen in an independent Java demonstration. The following curl inputs demonstrate my problem. curl -XPUT '[http://localhost:9200/twitter\_jp](http://localhost:9200/twitter_jp)' -d '{ "index" : { "numberOfShards" : 1, "numberOfReplicas" : 1, "analysis" : { "analyzer" : { "my\_analyzer" : { "type" : "cjk" } } } } }' curl -XPUT '[http://localhost:9200/twitter\_jp/tweet/1](http://localhost:9200/twitter_jp/tweet/1)' -d '{ "user": "wolfkden", "postDate": "Tue Mar 8 22:56:57 PST 2011", "text": "トヨタ自動車は、ハイブリッド車（ＨＶ）「プリウス」のワゴン型試作車「プリウス・スペースコンセプト」を国内で初めて公開 した。" }' curl -XPUT '[http://localhost:9200/twitter\_jp/tweet/2](http://localhost:9200/twitter_jp/tweet/2)' -d '{ "user": "wolfkden", "postDate": "Tue Mar 8 22:56:57 PST 2011", "text": "自動車メーカーで生産世界首位。国内販売で「プリウス」が好調も、世界的には大規模リコール問題での信頼回復が急務" }' curl -XGET '[http://localhost:9200/twitter\_jp/tweet/\_search](http://localhost:9200/twitter_jp/tweet/_search)' -d '{ "query" : { "match\_all" : { } }, "facets" : { "tag" : { "terms" : { "field" : "text", "size" : 30 } } } }' the output provides counts for individual characters rather than words: "facets":{"tag":{"\_type":"terms","missing":0,"terms": [{"term":"車","count":2},{"term":"自","count":2},{"term":"国","count":2}, {"term":"動","count":2},{"term":"内","count":2},{"term":"ー","count":2}, {"term":"リ","count":2},{"term":"プ","count":2},{"term":"ス","count":2}, {"term":"コ","count":2},{"term":"ウ","count":2},{"term":"は","count":2}, {"term":"の","count":2},{"term":"で","count":2},{"term":"ｈｖ","count":1}, {"term":"首","count":1},{"term":"題","count":1},{"term":"頼","count":1}, {"term":"開","count":1},{"term":"販","count":1},{"term":"調","count":1}, {"term":"試","count":1},{"term":"規","count":1},{"term":"的","count":1}, {"term":"界","count":1},{"term":"産","count":1},{"term":"生","count":1}, {"term":"模","count":1},{"term":"急","count":1},{"term":"復","count":1}, {"term":"好","count":1},{"term":"大","count":1},{"term":"売","count":1}, {"term":"型","count":1},{"term":"回","count":1},{"term":"問","count":1}, {"term":"務","count":1},{"term":"初","count":1},{"term":"公","count":1}, {"term":"信","count":1}]}}} In fact the word プリウス (Prius for toyota Prius) occurs in one of the documents. I only expect the sen software to pick this prius test out but the cjk analyzer should tokenize the two character words correctly. I checked the elasticsearch code and noted that the cjk analyzer type is instanced through the CjkAnalysisProvider in the AnalysisModule.processAnalyzer method. In the IndicesAnalysisService the cjk setting loads the ChineseAnalyzer rather than the CJKAnalyzer, I thought this was a problem in the code and made the correction but I still have the same problem. All in all though elasticsearch still does not tokenize east asian words correctly. Presently my efforts are focusing on the fact that the token stream from the analyzer is not used and hence the CjK analyzer is not implemented although the bindings are set. Could I get some more insight into this to solve my problem. Regards, Wolf ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [March 10, 2011, 6:19am UTC](https://discuss.elastic.co/t/asian-characters-and-not-words-are-tokenized-cjk-analysis-and-tokenization-problems/4072/2 "2011-03-10T06:19:53Z")
I did not see in where you created mappings that marked the CJK analyzer as either the default analyzer or the analyzer for text. On Thursday, March 10, 2011 at 6:00 AM, Wolf wrote: > I am using the CJK Analyzer to index Japanese language content into > elasticsearch: However the the input is not tokenized into words but > rather characters. The CJK analyzer does not have this problem when > using a stand alone lucene demonstration program, pp. 147, "Lucene in > Action" second edition, Manning 2010. I also do not have any problems > with Sen in an independent Java demonstration. The following curl > inputs demonstrate my problem. > > curl -XPUT '[http://localhost:9200/twitter\_jp](http://localhost:9200/twitter_jp)' -d '{ > "index" : { > "numberOfShards" : 1, > "numberOfReplicas" : 1, > "analysis" : { > "analyzer" : { > "my\_analyzer" : { > "type" : "cjk" > } > } > } > } > }' > > curl -XPUT '[http://localhost:9200/twitter\_jp/tweet/1](http://localhost:9200/twitter_jp/tweet/1)' -d '{ > "user": "wolfkden", > "postDate": "Tue Mar 8 22:56:57 PST 2011", > "text": "トヨタ自動車は、ハイブリッド車（ＨＶ）「プリウス」のワゴン型試作車「プリウス・スペースコンセプト」を国内で初めて公開 > した。" > }' > > curl -XPUT '[http://localhost:9200/twitter\_jp/tweet/2](http://localhost:9200/twitter_jp/tweet/2)' -d '{ > "user": "wolfkden", > "postDate": "Tue Mar 8 22:56:57 PST 2011", > "text": "è‡ªå‹•è»Šãƒ¡ãƒ¼ã‚«ãƒ¼ã§ç”Ÿç”£ä¸–ç•Œé¦–ä½ã€‚å›½å†…è²©å£²ã§ã€Œãƒ—ãƒªã‚¦ã‚¹ã€ãŒå¥½èª¿ã‚‚ã€ä¸–ç•Œçš„ã«ã¯å¤§è¦æ¨¡ãƒªã‚³ãƒ¼ãƒ«å•é¡Œã§ã®ä¿¡é ¼å›žå¾©ãŒæ€¥å‹™" > }' > > curl -XGET '[http://localhost:9200/twitter\_jp/tweet/\_search](http://localhost:9200/twitter_jp/tweet/_search)' -d '{ > "query" : { > "match\_all" : { } > }, > "facets" : { > "tag" : { > "terms" : { > "field" : "text", > "size" : 30 > } > } > } > }' > > the output provides counts for individual characters rather than > words: > > "facets":{"tag":{"\_type":"terms","missing":0,"terms": > [{"term":"車","count":2},{"term":"自","count":2},{"term":"国","count":2}, > {"term":"動","count":2},{"term":"内","count":2},{"term":"ー","count":2}, > {"term":"リ","count":2},{"term":"プ","count":2},{"term":"ス","count":2}, > {"term":"コ","count":2},{"term":"ウ","count":2},{"term":"は","count":2}, > {"term":"の","count":2},{"term":"で","count":2},{"term":"ｈｖ","count":1}, > {"term":"首","count":1},{"term":"題","count":1},{"term":"é ¼","count":1}, > {"term":"開","count":1},{"term":"販","count":1},{"term":"調","count":1}, > {"term":"試","count":1},{"term":"規","count":1},{"term":"的","count":1}, > {"term":"界","count":1},{"term":"産","count":1},{"term":"生","count":1}, > {"term":"模","count":1},{"term":"急","count":1},{"term":"復","count":1}, > {"term":"好","count":1},{"term":"大","count":1},{"term":"売","count":1}, > {"term":"型","count":1},{"term":"回","count":1},{"term":"問","count":1}, > {"term":"務","count":1},{"term":"初","count":1},{"term":"公","count":1}, > {"term":"ä¿¡","count":1}]}}} > > In fact the word プリウス (Prius for toyota Prius) occurs in one of the > documents. I only expect the sen software to pick this prius test out > but the cjk analyzer should tokenize the two character words > correctly. > > I checked the elasticsearch code and noted that the cjk analyzer type > is instanced through the CjkAnalysisProvider in the > AnalysisModule.processAnalyzer method. In the IndicesAnalysisService > the cjk setting loads the ChineseAnalyzer rather than the CJKAnalyzer, > I thought this was a problem in the code and made the correction but I > still have the same problem. > > All in all though elasticsearch still does not tokenize east asian > words correctly. > > Presently my efforts are focusing on the fact that the token stream > from the analyzer is not used and hence the CjK analyzer is not > implemented although the bindings are set. > > Could I get some more insight into this to solve my problem. > > Regards, > > Wolf ---
**Author:** ![Wolf\_2](https://avatars.discourse-cdn.com/v4/letter/w/ac8455/32.png) [@Wolf\_2](https://discuss.elastic.co/u/Wolf_2)\ **Post date:** [March 10, 2011, 9:21am UTC](https://discuss.elastic.co/t/asian-characters-and-not-words-are-tokenized-cjk-analysis-and-tokenization-problems/4072/3 "2011-03-10T09:21:30Z")
Thank you Shay for your quick response and personal consideration. I noted your comment on the mapping and followed up with a mapping declaration after the index declaration: curl -XPUT '[http://localhost:9200/twitter\_jp/\_mapping](http://localhost:9200/twitter_jp/_mapping)' -d '{ "tweet": { "properties": { "user":{"type":"string"}, "postDate":{"type":"string"}, "text": { "type": "string", "index": "analyzed", "analyzer" : "cjk" } } } }' I also executed the text field mapping with "search\_analyzer" : "cjk" and with "index": "not\_analyzed", "search\_analyzer": "cjk" switching off the indexing to not\_analyzed caused no tokenization to occur with the facet response being two terms one for each document. The interpreter is reading the cjk property since if I make a mispelling I get a guice error. these results used version 0.15.2. I also compiled a new version of Elasticsearch to use the CJKAnalyzer for binding against the cjk parameter in the IndicesAnalysisService class. This version produced the same results "I used guice comments to verify that the binding code executed". I am still concerned that I cannot find where the CJKAnalyzer.tokenStream is referenced in the code. Possibly the default "standard" tokenizer is being used. Please let me know what else is possibly wrong with my scripts or other remedies that I can pursue. Regards, Wolf On Wed, Mar 9, 2011 at 10:19 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > I did not see in where you created mappings that marked the CJK analyzer > as either the default analyzer or the analyzer for text. > > On Thursday, March 10, 2011 at 6:00 AM, Wolf wrote: > > I am using the CJK Analyzer to index Japanese language content into > elasticsearch: However the the input is not tokenized into words but > rather characters. The CJK analyzer does not have this problem when > using a stand alone lucene demonstration program, pp. 147, "Lucene in > Action" second edition, Manning 2010. I also do not have any problems > with Sen in an independent Java demonstration. The following curl > inputs demonstrate my problem. > > curl -XPUT '[http://localhost:9200/twitter\_jp](http://localhost:9200/twitter_jp)' -d '{ > "index" : { > "numberOfShards" : 1, > "numberOfReplicas" : 1, > "analysis" : { > "analyzer" : { > ""my\_analyzer" : { > "type" : "cjk" > } > } > } > } > }' > > curl -XPUT '[http://localhost:9200/twitter\_jp/tweet/1](http://localhost:9200/twitter_jp/tweet/1)' -d '{ > "user": "wolfkden", > "postDate": "Tue Mar 8 22:56:57 PST 2011", > "text": "トヨタ自動車は、ハイブリッド車（ＨＶ）「プリウス」のワゴン型試作車「プリウス・スペースコンセプト」を国内で初めて公開 > した。" > }' > > curl -XPUT '[http://localhost:9200/twitter\_jp/tweet/2](http://localhost:9200/twitter_jp/tweet/2)' -d '{ > "user": "wolfkden", > "postDate": "Tue Mar 8 22:56:57 PST 2011", > "text": "自動車メーカーで生産世界首位。国内販売で「プリウス」が好調も、世界的には大規模リコール問題での信頼回復が急務" > }' > > curl -XGET '[http://localhost:9200/twitter\_jp/tweet/\_search](http://localhost:9200/twitter_jp/tweet/_search)' -d '{ > "query" : { > "match\_all" : { }} > }, > "facets" : { > "tag" : { > "terms" : { > "field" : "text", > "size" : 30 > } > } > } > }' > > the output provides counts for individual characters rather than > words: > > "facets":{"tag":{"\_type":"terms","missing":0,"terms": > [{"term":"車","count":2},{"term":"自","count":2},{"term":"国","count":2}, > {"term":"動","count":2},{"term":"内","count":2},{"term":"ー","count":2}, > {"term":"リ","count":2},{"term":"プ","count":2},{"term":"ス","count":2}, > {"term":"コ","count":2},{"term":"ウ","count":2},{"term":"は","count":2}, > {"term":"の","count":2},{"term":"で","count":2},{"term":"ｈｖ","count":1}, > {"term":"首","count":1},{"term":"題","count":1},{"term":"頼","count":1}, > {"term":"開","count":1},{"term":"販","count":1},{"term":"調","count":1}, > {"term":"試","count":1},{"term":"規","count":1},{"term":"的","count":1}, > {"term":"界","count":1},{"term":"産","count":1},{"term":"生","count":1}, > {"term":"模","count":1},{"term":"急","count":1},{"term":"復","count":1}, > {"term":"好","count":1},{"term":"大","count":1},{"term":"売","count":1}, > {"term":"型","count":1},{"term":"回","count":1},{"term":"問","count":1}, > {"term":"務","count":1},{"term":"初","count":1},{"term":"公","count":1}, > {"term":"信","count":1}]}}} > > In fact the word プリウス (Prius for toyota Prius) occurs in one of the > documents. I only expect the sen software to pick this prius test out > but the cjk analyzer should tokenize the two character words > correctly. > > I checked the elasticsearch code and noted that the cjk analyzer type > is instanced through the CjkAnalysisProvider in the > AnalysisModule.processAnalyzer method. In the IndicesAnalysisService > the cjk setting loads the ChineseAnalyzer rather than the CJKAnalyzer, > I thought this was a problem in the code and made the correction but I > still have the same problem. > > All in all though elasticsearch still does not tokenize east asian > words correctly. > > Presently my efforts are focusing on the fact that the token stream > from the analyzer is not used and hence the CjK analyzer is not > implemented although the bindings are set. > > Could I get some more insight into this to solve my problem. > > Regards, > > Wolf ---
**Author:** ![Wolf\_2](https://avatars.discourse-cdn.com/v4/letter/w/ac8455/32.png) [@Wolf\_2](https://discuss.elastic.co/u/Wolf_2)\ **Post date:** [March 10, 2011, 6:34pm UTC](https://discuss.elastic.co/t/asian-characters-and-not-words-are-tokenized-cjk-analysis-and-tokenization-problems/4072/4 "2011-03-10T18:34:25Z")
Thank you Shay for your quick response and personal consideration. I noted your comment on the mapping and followed up with a mapping declaration after the index declaration: curl -XPUT '[http://localhost:9200/twitter\_jp/\_mapping](http://localhost:9200/twitter_jp/_mapping)' -d '{ "tweet": { "properties": { "user":{"type":"string"}, "postDate":{"type":"string"}, "text": { "type": "string", "index": "analyzed", "analyzer" : "cjk" } } } }' I also executed the text field mapping with "search\_analyzer" : "cjk" and with "index": "not\_analyzed", "search\_analyzer": "cjk" switching off the indexing to not\_analyzed caused no tokenization to occur with the facet response being two terms one for each document. The interpreter is reading the cjk property since if I make a mispelling I get a guice error. these results used version 0.15.2. I also compiled a new version of Elasticsearch to use the CJKAnalyzer for binding against the cjk parameter in the IndicesAnalysisService class. This version produced the same results "I used guice comments to verify that the binding code executed". I am still concerned that I cannot find where the CJKAnalyzer.tokenStream is referenced in the code. Possibly the default "standard" tokenizer is being used. Please let me know what else is possibly wrong with my scripts or other remedies that I can pursue. Regards, Wolf On Mar 9, 8:00 pm, Wolf [wolfk...@gmail.com](mailto:wolfk...@gmail.com) wrote: > I am using the CJK Analyzer to index Japanese language content into > elasticsearch: However the the input is not tokenized into words but > rather characters. The CJK analyzer does not have this problem when > using a stand alone lucene demonstration program, pp. 147, "Lucene in > Action" second edition, Manning 2010. I also do not have any problems > with Sen in an independent Java demonstration. The following curl > inputs demonstrate my problem. > > curl -XPUT '[http://localhost:9200/twitter\_jp'-d](http://localhost:9200/twitter_jp'-d) '{ > "index" : { > "numberOfShards" : 1, > "numberOfReplicas" : 1, > "analysis" : { > "analyzer" : { > "my\_analyzer" : { > "type" : "cjk" > } > } > } > } > > }' > > curl -XPUT '[http://localhost:9200/twitter\_jp/tweet/1'-d](http://localhost:9200/twitter_jp/tweet/1'-d) '{ > "user": "wolfkden", > "postDate": "Tue Mar 8 22:56:57 PST 2011", > "text": "トヨタ自動車は、ハイブリッド車（ＨＶ）「プリウス」のワゴン型試作車「プリウス・スペースコンセプト」を国内で初めて公開 > した。" > > }' > > curl -XPUT '[http://localhost:9200/twitter\_jp/tweet/2'-d](http://localhost:9200/twitter_jp/tweet/2'-d) '{ > "user": "wolfkden", > "postDate": "Tue Mar 8 22:56:57 PST 2011", > "text": "自動車メーカーで生産世界首位。国内販売で「プリウス」が好調も、世界的には大規模リコール問題での信頼回復が急務" > > }' > > curl -XGET '[http://localhost:9200/twitter\_jp/tweet/\_search'-d](http://localhost:9200/twitter_jp/tweet/_search'-d) '{ > "query" : { > "match\_all" : { } > }, > "facets" : { > "tag" : { > "terms" : { > "field" : "text", > "size" : 30 > } > } > } > > }' > > the output provides counts for individual characters rather than > words: > > "facets":{"tag":{"\_type":"terms","missing":0,"terms": > [{"term":"車","count":2},{"term":"自","count":2},{"term":"国","count":2}, > {"term":"動","count":2},{"term":"内","count":2},{"term":"ー","count":2}, > {"term":"リ","count":2},{"term":"プ","count":2},{"term":"ス","count":2}, > {"term":"コ","count":2},{"term":"ウ","count":2},{"term":"は","count":2}, > {"term":"の","count":2},{"term":"で","count":2},{"term":"ｈｖ","count":1}, > {"term":"首","count":1},{"term":"題","count":1},{"term":"頼","count":1}, > {"term":"開","count":1},{"term":"販","count":1},{"term":"調","count":1}, > {"term":"試","count":1},{"term":"規","count":1},{"term":"的","count":1}, > {"term":"界","count":1},{"term":"産","count":1},{"term":"生","count":1}, > {"term":"模","count":1},{"term":"急","count":1},{"term":"復","count":1}, > {"term":"好","count":1},{"term":"大","count":1},{"term":"売","count":1}, > {"term":"型","count":1},{"term":"回","count":1},{"term":"問","count":1}, > {"term":"務","count":1},{"term":"初","count":1},{"term":"公","count":1}, > {"term":"信","count":1}]}}} > > In fact the word プリウス (Prius for toyota Prius) occurs in one of the > documents. I only expect the sen software to pick this prius test out > but the cjk analyzer should tokenize the two character words > correctly. > > I checked the elasticsearch code and noted that the cjk analyzer type > is instanced through the CjkAnalysisProvider in the > AnalysisModule.processAnalyzer method. In the IndicesAnalysisService > the cjk setting loads the ChineseAnalyzer rather than the CJKAnalyzer, > I thought this was a problem in the code and made the correction but I > still have the same problem. > > All in all though elasticsearch still does not tokenize east asian > words correctly. > > Presently my efforts are focusing on the fact that the token stream > from the analyzer is not used and hence the CjK analyzer is not > implemented although the bindings are set. > > Could I get some more insight into this to solve my problem. > > Regards, > > Wolf ---
**Author:** ![Wolf\_2](https://avatars.discourse-cdn.com/v4/letter/w/ac8455/32.png) [@Wolf\_2](https://discuss.elastic.co/u/Wolf_2)\ **Post date:** [March 10, 2011, 7:53pm UTC](https://discuss.elastic.co/t/asian-characters-and-not-words-are-tokenized-cjk-analysis-and-tokenization-problems/4072/5 "2011-03-10T19:53:58Z")
Thank you Shay for your quick response and personal consideration. I added more specif analysis mapping for the text field but problems persist. I noted your comment on the mapping and followed up with a mapping declaration after the index declaration: curl -XPUT '[http://localhost:9200/twitter\_jp/\_mapping](http://localhost:9200/twitter_jp/_mapping)' -d '{ "tweet": { "properties": { "user":{"type":"string"}, "postDate":{"type":"string"}, "text": { "type": "string", "index": "analyzed", "analyzer" : "cjk" } } } }' I also executed the text field mapping with "search\_analyzer" : "cjk" and with "index": "not\_analyzed", "search\_analyzer": "cjk" switching off the indexing to not\_analyzed caused no tokenization to occur with the facet response being two terms one for each document. The interpreter is reading the cjk property since if I make a mispelling I get a guice error. these results used version 0.15.2. I also compiled a new version of Elasticsearch to use the CJKAnalyzer for binding against the cjk parameter in the IndicesAnalysisService class. This version produced the same results "I used guice comments to verify that the binding code executed". I am still concerned that I cannot find where the CJKAnalyzer.tokenStream is referenced in the code. Possibly the default "standard" tokenizer is being used. Please let me know what else is possibly wrong with my scripts or other remedies that I can pursue. Regards, Wolf On Mar 9, 8:00 pm, Wolf [wolfk...@gmail.com](mailto:wolfk...@gmail.com) wrote: > I am using the CJK Analyzer to index Japanese language content into > elasticsearch: However the the input is not tokenized into words but > rather characters. The CJK analyzer does not have this problem when > using a stand alone lucene demonstration program, pp. 147, "Lucene in > Action" second edition, Manning 2010. I also do not have any problems > with Sen in an independent Java demonstration. The following curl > inputs demonstrate my problem. > > curl -XPUT '[http://localhost:9200/twitter\_jp'-d](http://localhost:9200/twitter_jp'-d) '{ > "index" : { > "numberOfShards" : 1, > "numberOfReplicas" : 1, > "analysis" : { > "analyzer" : { > "my\_analyzer" : { > "type" : "cjk" > } > } > } > } > > }' > > curl -XPUT '[http://localhost:9200/twitter\_jp/tweet/1'-d](http://localhost:9200/twitter_jp/tweet/1'-d) '{ > "user": "wolfkden", > "postDate": "Tue Mar 8 22:56:57 PST 2011", > "text": "トヨタ自動車は、ハイブリッド車（ＨＶ）「プリウス」のワゴン型試作車「プリウス・スペースコンセプト」を国内で初めて公開 > した。" > > }' > > curl -XPUT '[http://localhost:9200/twitter\_jp/tweet/2'-d](http://localhost:9200/twitter_jp/tweet/2'-d) '{ > "user": "wolfkden", > "postDate": "Tue Mar 8 22:56:57 PST 2011", > "text": "自動車メーカーで生産世界首位。国内販売で「プリウス」が好調も、世界的には大規模リコール問題での信頼回復が急務" > > }' > > curl -XGET '[http://localhost:9200/twitter\_jp/tweet/\_search'-d](http://localhost:9200/twitter_jp/tweet/_search'-d) '{ > "query" : { > "match\_all" : { } > }, > "facets" : { > "tag" : { > "terms" : { > "field" : "text", > "size" : 30 > } > } > } > > }' > > the output provides counts for individual characters rather than > words: > > "facets":{"tag":{"\_type":"terms","missing":0,"terms": > [{"term":"車","count":2},{"term":"自","count":2},{"term":"国","count":2}, > {"term":"動","count":2},{"term":"内","count":2},{"term":"ー","count":2}, > {"term":"リ","count":2},{"term":"プ","count":2},{"term":"ス","count":2}, > {"term":"コ","count":2},{"term":"ウ","count":2},{"term":"は","count":2}, > {"term":"の","count":2},{"term":"で","count":2},{"term":"ｈｖ","count":1}, > {"term":"首","count":1},{"term":"題","count":1},{"term":"頼","count":1}, > {"term":"開","count":1},{"term":"販","count":1},{"term":"調","count":1}, > {"term":"試","count":1},{"term":"規","count":1},{"term":"的","count":1}, > {"term":"界","count":1},{"term":"産","count":1},{"term":"生","count":1}, > {"term":"模","count":1},{"term":"急","count":1},{"term":"復","count":1}, > {"term":"好","count":1},{"term":"大","count":1},{"term":"売","count":1}, > {"term":"型","count":1},{"term":"回","count":1},{"term":"問","count":1}, > {"term":"務","count":1},{"term":"初","count":1},{"term":"公","count":1}, > {"term":"信","count":1}]}}} > > In fact the word プリウス (Prius for toyota Prius) occurs in one of the > documents. I only expect the sen software to pick this prius test out > but the cjk analyzer should tokenize the two character words > correctly. > > I checked the elasticsearch code and noted that the cjk analyzer type > is instanced through the CjkAnalysisProvider in the > AnalysisModule.processAnalyzer method. In the IndicesAnalysisService > the cjk setting loads the ChineseAnalyzer rather than the CJKAnalyzer, > I thought this was a problem in the code and made the correction but I > still have the same problem. > > All in all though elasticsearch still does not tokenize east asian > words correctly. > > Presently my efforts are focusing on the fact that the token stream > from the analyzer is not used and hence the CjK analyzer is not > implemented although the bindings are set. > > Could I get some more insight into this to solve my problem. > > Regards, > > Wolf ---
**Author:** ![cwho](https://avatars.discourse-cdn.com/v4/letter/c/b5ac83/32.png) [@cwho](https://discuss.elastic.co/u/cwho)\ **Post date:** [March 11, 2011, 6:03am UTC](https://discuss.elastic.co/t/asian-characters-and-not-words-are-tokenized-cjk-analysis-and-tokenization-problems/4072/6 "2011-03-11T06:03:51Z")
The prebuilt "cjk" analyzer does indeed use ChineseAnalyzer instead of CJKAnalyzer. Instead of using the prebuilt "cjk", try defining a new analyzer "my\_cjk" in your configuration pointing to CjkAnalyzerProvider instead: index: analysis: analyzer: my\_cjk: type: org.elasticsearch.index.analysis.CjkAnalyzerProvider Then following your example, it appears to be working for me - now doing the bigram character words that the CJKAnalyzer uses. I'm using 0.15.1. curl -XPUT '[http://localhost:9200/twitter\_jp](http://localhost:9200/twitter_jp)' -d '{ "index" : { "numberOfShards" : 1, "numberOfReplicas" : 1 } }' curl -XPUT '[http://localhost:9200/twitter\_jp/\_mapping](http://localhost:9200/twitter_jp/_mapping)' -d '{ "tweet": { "properties": { "user":{"type":"string"}, "postDate":{"type":"string"}, "text": { "type": "string", "index": "analyzed", "analyzer" : "my\_cjk" } } } }' curl -XPUT '[http://localhost:9200/twitter\_jp/tweet/1](http://localhost:9200/twitter_jp/tweet/1)' -d '{ "user": "wolfkden", "postDate": "Tue Mar 8 22:56:57 PST 2011", "text": "トヨタ自動車は、ハイブリッド車（ＨＶ）「プリウス」のワゴン型試作車「プリウス・スペースコンセプト」を国内で初めて公開 した。" }' curl -XGET '[http://localhost:9200/twitter\_jp/tweet/\_search](http://localhost:9200/twitter_jp/tweet/_search)' -d '{ "query" : { "match\_all" : { } }, "facets" : { "tag" : { "terms" : { "field" : "text", "size" : 30 } } } }' {"took":182,"timed\_out":false,"\_shards":{"total":1,"successful": 1,"failed":0},"hits":{"total":1,"max\_score":1.0,"hits": [{"\_index":"twitter\_jp","\_type":"tweet","\_id":"1","\_score":1.0, "\_source" : { "user": "wolfkden", "postDate": "Tue Mar 8 22:56:57 PST 2011", "text": "トヨタ自動車は、ハイブリッド車（ＨＶ）「プリウス」のワゴン型試作車「プリウス・スペースコンセプト」を国内で初めて公開 した。" }}]},"facets":{"tag":{"\_type":"terms","missing":0,"terms":[{"term":"開 し","count":1},{"term":"車は","count":1},{"term":"試作","count":1}, {"term":"自動","count":1},{"term":"型試","count":1},{"term":"国内","count": 1},{"term":"動車","count":1},{"term":"初め","count":1},{"term":"内 で","count":1},{"term":"公開","count":1},{"term":"作車","count":1}, {"term":"ース","count":1},{"term":"ン型","count":1},{"term":"ンセ","count": 1},{"term":"ワゴ","count":1},{"term":"リッ","count":1},{"term":"リ ウ","count":1},{"term":"ヨタ","count":1},{"term":"ペー","count":1}, {"term":"プリ","count":1},{"term":"プト","count":1},{"term":"ブリ","count": 1},{"term":"ハイ","count":1},{"term":"ド車","count":1},{"term":"ト ヨ","count":1},{"term":"ッド","count":1},{"term":"タ自","count":1}, {"term":"セプ","count":1},{"term":"スペ","count":1},{"term":"スコ","count": 1}]}}} Hope this helps. -C On Mar 11, 3:53 am, Wolf [wolfk...@gmail.com](mailto:wolfk...@gmail.com) wrote: > Thank you Shay for your quick response and personal consideration. > > I added more specif analysis mapping for the text field but problems > persist. > > I noted your comment on the mapping and followed up with a mapping > declaration after the index declaration: > > curl -XPUT '[http://localhost:9200/twitter\_jp/\_mapping'-d](http://localhost:9200/twitter_jp/_mapping'-d) '{ > "tweet": { > "properties": { > "user":{"type":"string"}, > "postDate":{"type":"string"}, > "text": { "type": "string", "index": "analyzed", > "analyzer" : > "cjk" } > } > } > > }' > > I also executed the text field mapping with "search\_analyzer" : "cjk" > and > with > "index": "not\_analyzed", "search\_analyzer": "cjk" > > switching off the indexing to not\_analyzed caused no tokenization to > occur > with the facet response being two terms one for each document. The > interpreter is reading the cjk property since if I make a mispelling I > get a > guice error. > > these results used version 0.15.2. > > I also compiled a new version of Elasticsearch to use the CJKAnalyzer > for > binding against the cjk parameter in the IndicesAnalysisService class. > This > version produced the same results "I used guice comments to verify > that the > binding code executed". > > I am still concerned that I cannot find where the > CJKAnalyzer.tokenStream is > referenced in the code. Possibly the default "standard" tokenizer is > being > used. > > Please let me know what else is possibly wrong with my scripts or > other > remedies that I can pursue. > > Regards, > > Wolf > > On Mar 9, 8:00 pm, Wolf [wolfk...@gmail.com](mailto:wolfk...@gmail.com) wrote: > > > I am using the CJK Analyzer to index Japanese language content into > > elasticsearch: However the the input is not tokenized into words but > > rather characters. The CJK analyzer does not have this problem when > > using a stand alone lucene demonstration program, pp. 147, "Lucene in > > Action" second edition, Manning 2010. I also do not have any problems > > with Sen in an independent Java demonstration. The following curl > > inputs demonstrate my problem. > > > curl -XPUT '[http://localhost:9200/twitter\_jp'-d'](http://localhost:9200/twitter_jp'-d'){ > > "index" : { > > "numberOfShards" : 1, > > "numberOfReplicas" : 1, > > "analysis" : { > > "analyzer" : { > > "my\_analyzer" : { > > "type" : "cjk" > > } > > } > > } > > } > > > }' > > > curl -XPUT '[http://localhost:9200/twitter\_jp/tweet/1'-d'](http://localhost:9200/twitter_jp/tweet/1'-d'){ > > "user": "wolfkden", > > "postDate": "Tue Mar 8 22:56:57 PST 2011", > > "text": "トヨタ自動車は、ハイブリッド車（ＨＶ）「プリウス」のワゴン型試作車「プリウス・スペースコンセプト」を国内で初めて公開 > > した。" > > > }' > > > curl -XPUT '[http://localhost:9200/twitter\_jp/tweet/2'-d'](http://localhost:9200/twitter_jp/tweet/2'-d'){ > > "user": "wolfkden", > > "postDate": "Tue Mar 8 22:56:57 PST 2011", > > "text": "自動車メーカーで生産世界首位。国内販売で「プリウス」が好調も、世界的には大規模リコール問題での信頼回復が急務" > > > }' > > > curl -XGET '[http://localhost:9200/twitter\_jp/tweet/\_search'-d'](http://localhost:9200/twitter_jp/tweet/_search'-d'){ > > "query" : { > > "match\_all" : { } > > }, > > "facets" : { > > "tag" : { > > "terms" : { > > "field" : "text", > > "size" : 30 > > } > > } > > } > > > }' > > > the output provides counts for individual characters rather than > > words: > > > "facets":{"tag":{"\_type":"terms","missing":0,"terms": > > [{"term":"車","count":2},{"term":"自","count":2},{"term":"国","count":2}, > > {"term":"動","count":2},{"term":"内","count":2},{"term":"ー","count":2}, > > {"term":"リ","count":2},{"term":"プ","count":2},{"term":"ス","count":2}, > > {"term":"コ","count":2},{"term":"ウ","count":2},{"term":"は","count":2}, > > {"term":"の","count":2},{"term":"で","count":2},{"term":"ｈｖ","count":1}, > > {"term":"首","count":1},{"term":"題","count":1},{"term":"頼","count":1}, > > {"term":"開","count":1},{"term":"販","count":1},{"term":"調","count":1}, > > {"term":"試","count":1},{"term":"規","count":1},{"term":"的","count":1}, > > {"term":"界","count":1},{"term":"産","count":1},{"term":"生","count":1}, > > {"term":"模","count":1},{"term":"急","count":1},{"term":"復","count":1}, > > {"term":"好","count":1},{"term":"大","count":1},{"term":"売","count":1}, > > {"term":"型","count":1},{"term":"回","count":1},{"term":"問","count":1}, > > {"term":"務","count":1},{"term":"初","count":1},{"term":"公","count":1}, > > {"term":"信","count":1}]}}} > > > In fact the word プリウス (Prius for toyota Prius) occurs in one of the > > documents. I only expect the sen software to pick this prius test out > > but the cjk analyzer should tokenize the two character words > > correctly. > > > I checked the elasticsearch code and noted that the cjk analyzer type > > is instanced through the CjkAnalysisProvider in the > > AnalysisModule.processAnalyzer method. In the IndicesAnalysisService > > the cjk setting loads the ChineseAnalyzer rather than the CJKAnalyzer, > > I thought this was a problem in the code and made the correction but I > > still have the same problem. > > > All in all though elasticsearch still does not tokenize east asian > > words correctly. > > > Presently my efforts are focusing on the fact that the token stream > > from the analyzer is not used and hence the CjK analyzer is not > > implemented although the bindings are set. > > > Could I get some more insight into this to solve my problem. > > > Regards, > > > Wolf ---
**Author:** ![Wolf](https://avatars.discourse-cdn.com/v4/letter/w/e19adc/32.png) [@Wolf](https://discuss.elastic.co/u/Wolf)\ **Post date:** [March 11, 2011, 9:40am UTC](https://discuss.elastic.co/t/asian-characters-and-not-words-are-tokenized-cjk-analysis-and-tokenization-problems/4072/7 "2011-03-11T09:40:12Z")
Thank you cwho, you are most excellent. I tried the mod you suggested and I got the same result in version 0.15.2. I will also follow your lead on adding new custom analyzers from previous posts. Regards, Wolf ---
**Author:** ![Wolf\_2](https://avatars.discourse-cdn.com/v4/letter/w/ac8455/32.png) [@Wolf\_2](https://discuss.elastic.co/u/Wolf_2)\ **Post date:** [March 11, 2011, 9:42am UTC](https://discuss.elastic.co/t/asian-characters-and-not-words-are-tokenized-cjk-analysis-and-tokenization-problems/4072/8 "2011-03-11T09:42:20Z")
Thank you cwho, you are most excellent. I tried the mod you suggested and I got the same result in version 0.15.2. I will also follow your lead on adding new custom analyzers from previous posts. Regards, Wolf On Mar 10, 10:03 pm, cwho80 [fuzzyb...@gmail.com](mailto:fuzzyb...@gmail.com) wrote: > The prebuilt "cjk" analyzer does indeed use ChineseAnalyzer instead of > CJKAnalyzer. > > Instead of using the prebuilt "cjk", try defining a new analyzer > "my\_cjk" in your configuration pointing to CjkAnalyzerProvider > instead: > > index: > analysis: > analyzer: > my\_cjk: > type: > org.elasticsearch.index.analysis.CjkAnalyzerProvider > > Then following your example, it appears to be working for me - now > doing the bigram character words that the CJKAnalyzer uses. I'm using > 0.15.1. > > curl -XPUT '[http://localhost:9200/twitter\_jp'-d](http://localhost:9200/twitter_jp'-d) '{ > "index" : { > "numberOfShards" : 1, > "numberOfReplicas" : 1 > } > > }' > > curl -XPUT '[http://localhost:9200/twitter\_jp/\_mapping'-d](http://localhost:9200/twitter_jp/_mapping'-d) '{ > "tweet": { > "properties": { > "user":{"type":"string"}, > "postDate":{"type":"string"}, > "text": { "type": "string", "index": "analyzed", > "analyzer" : "my\_cjk" } > } > } > > }' > > curl -XPUT '[http://localhost:9200/twitter\_jp/tweet/1'-d](http://localhost:9200/twitter_jp/tweet/1'-d) '{ > "user": "wolfkden", > "postDate": "Tue Mar 8 22:56:57 PST 2011", > "text": "トヨタ自動車は、ハイブリッド車（ＨＶ）「プリウス」のワゴン型試作車「プリウス・スペースコンセプト」を国内で初めて公開 > した。" > > }' > > curl -XGET '[http://localhost:9200/twitter\_jp/tweet/\_search'-d](http://localhost:9200/twitter_jp/tweet/_search'-d) '{ > "query" : { > "match\_all" : { } > }, > "facets" : { > "tag" : { > "terms" : { > "field" : "text", > "size" : 30 > } > } > } > > }' > > {"took":182,"timed\_out":false,"\_shards":{"total":1,"successful": > 1,"failed":0},"hits":{"total":1,"max\_score":1.0,"hits": > [{"\_index":"twitter\_jp","\_type":"tweet","\_id":"1","\_score":1.0, > "\_source" : { > "user": "wolfkden", > "postDate": "Tue Mar 8 22:56:57 PST 2011", > "text": "トヨタ自動車は、ハイブリッド車（ＨＶ）「プリウス」のワゴン型試作車「プリウス・スペースコンセプト」を国内で初めて公開 > した。"}}]},"facets":{"tag":{"\_type":"terms","missing":0,"terms":[{"term":"開 > > し","count":1},{"term":"車は","count":1},{"term":"試作","count":1}, > {"term":"自動","count":1},{"term":"型試","count":1},{"term":"国内","count": > 1},{"term":"動車","count":1},{"term":"初め","count":1},{"term":"内 > で","count":1},{"term":"公開","count":1},{"term":"作車","count":1}, > {"term":"ース","count":1},{"term":"ン型","count":1},{"term":"ンセ","count": > 1},{"term":"ワゴ","count":1},{"term":"リッ","count":1},{"term":"リ > ウ","count":1},{"term":"ヨタ","count":1},{"term":"ペー","count":1}, > {"term":"プリ","count":1},{"term":"プト","count":1},{"term":"ブリ","count": > 1},{"term":"ハイ","count":1},{"term":"ド車","count":1},{"term":"ト > ヨ","count":1},{"term":"ッド","count":1},{"term":"タ自","count":1}, > {"term":"セプ","count":1},{"term":"スペ","count":1},{"term":"スコ","count": > 1}]}}} > > Hope this helps. > > -C > > On Mar 11, 3:53 am, Wolf [wolfk...@gmail.com](mailto:wolfk...@gmail.com) wrote: > > > Thank you Shay for your quick response and personal consideration. > > > I added more specif analysis mapping for the text field but problems > > persist. > > > I noted your comment on the mapping and followed up with a mapping > > declaration after the index declaration: > > > curl -XPUT '[http://localhost:9200/twitter\_jp/\_mapping'-d'](http://localhost:9200/twitter_jp/_mapping'-d'){ > > "tweet": { > > "properties": { > > "user":{"type":"string"}, > > "postDate":{"type":"string"}, > > "text": { "type": "string", "index": "analyzed", > > "analyzer" : > > "cjk" } > > } > > } > > > }' > > > I also executed the text field mapping with "search\_analyzer" : "cjk" > > and > > with > > "index": "not\_analyzed", "search\_analyzer": "cjk" > > > switching off the indexing to not\_analyzed caused no tokenization to > > occur > > with the facet response being two terms one for each document. The > > interpreter is reading the cjk property since if I make a mispelling I > > get a > > guice error. > > > these results used version 0.15.2. > > > I also compiled a new version of Elasticsearch to use the CJKAnalyzer > > for > > binding against the cjk parameter in the IndicesAnalysisService class. > > This > > version produced the same results "I used guice comments to verify > > that the > > binding code executed". > > > I am still concerned that I cannot find where the > > CJKAnalyzer.tokenStream is > > referenced in the code. Possibly the default "standard" tokenizer is > > being > > used. > > > Please let me know what else is possibly wrong with my scripts or > > other > > remedies that I can pursue. > > > Regards, > > > Wolf > > > On Mar 9, 8:00 pm, Wolf [wolfk...@gmail.com](mailto:wolfk...@gmail.com) wrote: > > > > I am using the CJK Analyzer to index Japanese language content into > > > elasticsearch: However the the input is not tokenized into words but > > > rather characters. The CJK analyzer does not have this problem when > > > using a stand alone lucene demonstration program, pp. 147, "Lucene in > > > Action" second edition, Manning 2010. I also do not have any problems > > > with Sen in an independent Java demonstration. The following curl > > > inputs demonstrate my problem. > > > > curl -XPUT '[http://localhost:9200/twitter\_jp'-d'](http://localhost:9200/twitter_jp'-d'){ > > > "index" : { > > > "numberOfShards" : 1, > > > "numberOfReplicas" : 1, > > > "analysis" : { > > > "analyzer" : { > > > "my\_analyzer" : { > > > "type" : "cjk" > > > } > > > } > > > } > > > } > > > > }' > > > > curl -XPUT '[http://localhost:9200/twitter\_jp/tweet/1'-d'](http://localhost:9200/twitter_jp/tweet/1'-d'){ > > > "user": "wolfkden", > > > "postDate": "Tue Mar 8 22:56:57 PST 2011", > > > "text": "トヨタ自動車は、ハイブリッド車（ＨＶ）「プリウス」のワゴン型試作車「プリウス・スペースコンセプト」を国内で初めて公開 > > > した。" > > > > }' > > > > curl -XPUT '[http://localhost:9200/twitter\_jp/tweet/2'-d'](http://localhost:9200/twitter_jp/tweet/2'-d'){ > > > "user": "wolfkden", > > > "postDate": "Tue Mar 8 22:56:57 PST 2011", > > > "text": "自動車メーカーで生産世界首位。国内販売で「プリウス」が好調も、世界的には大規模リコール問題での信頼回復が急務" > > > > }' > > > > curl -XGET '[http://localhost:9200/twitter\_jp/tweet/\_search'-d'](http://localhost:9200/twitter_jp/tweet/_search'-d'){ > > > "query" : { > > > "match\_all" : { } > > > }, > > > "facets" : { > > > "tag" : { > > > "terms" : { > > > "field" : "text", > > > "size" : 30 > > > } > > > } > > > } > > > > }' > > > > the output provides counts for individual characters rather than > > > words: > > > > "facets":{"tag":{"\_type":"terms","missing":0,"terms": > > > [{"term":"車","count":2},{"term":"自","count":2},{"term":"国","count":2}, > > > {"term":"動","count":2},{"term":"内","count":2},{"term":"ー","count":2}, > > > {"term":"リ","count":2},{"term":"プ","count":2},{"term":"ス","count":2}, > > > {"term":"コ","count":2},{"term":"ウ","count":2},{"term":"は","count":2}, > > > {"term":"の","count":2},{"term":"で","count":2},{"term":"ｈｖ","count":1}, > > > {"term":"首","count":1},{"term":"題","count":1},{"term":"頼","count":1}, > > > {"term":"開","count":1},{"term":"販","count":1},{"term":"調","count":1}, > > > {"term":"試","count":1},{"term":"規","count":1},{"term":"的","count":1}, > > > {"term":"界","count":1},{"term":"産","count":1},{"term":"生","count":1}, > > > {"term":"模","count":1},{"term":"急","count":1},{"term":"復","count":1}, > > > {"term":"好","count":1},{"term":"大","count":1},{"term":"売","count":1}, > > > {"term":"型","count":1},{"term":"回","count":1},{"term":"問","count":1}, > > > {"term":"務","count":1},{"term":"初","count":1},{"term":"公","count":1}, > > > {"term":"信","count":1}]}}} > > > > In fact the word プリウス (Prius for toyota Prius) occurs in one of the > > > documents. I only expect the sen software to pick this prius test out > > > but the cjk analyzer should tokenize the two character words > > > correctly. > > > > I checked the elasticsearch code and noted that the cjk analyzer type > > > is instanced through the CjkAnalysisProvider in the > > > AnalysisModule.processAnalyzer method. In the IndicesAnalysisService > > > the cjk setting loads the ChineseAnalyzer rather than the CJKAnalyzer, > > > I thought this was a problem in the code and made the correction but I > > > still have the same problem. > > > > All in all though elasticsearch still does not tokenize east asian > > > words correctly. > > > > Presently my efforts are focusing on the fact that the token stream > > > from the analyzer is not used and hence the CjK analyzer is not > > > implemented although the bindings are set. > > > > Could I get some more insight into this to solve my problem. > > > > Regards, > > > > Wolf ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:10am UTC](https://discuss.elastic.co/t/asian-characters-and-not-words-are-tokenized-cjk-analysis-and-tokenization-problems/4072/9 "2017-07-06T04:10:19Z")