# Whitespace tokenizer does not tokenise character code 160 **URL:** \ **Category:** Elasticsearch\ **Created:** [July 5, 2013, 10:06am UTC](https://discuss.elastic.co/t/whitespace-tokenizer-does-not-tokenise-character-code-160/12675 "2013-07-05T10:06:15Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![imdhmd](https://avatars.discourse-cdn.com/v4/letter/i/d07c76/32.png) [@imdhmd](https://discuss.elastic.co/u/imdhmd)\ **Post date:** [July 5, 2013, 10:06am UTC](https://discuss.elastic.co/t/whitespace-tokenizer-does-not-tokenise-character-code-160/12675/1 "2013-07-05T10:06:15Z")
Hello All, I'm using ES 0.20.6. (all the curl commands below are available at this gist: [https://gist.github.com/imdhmd/cda6880e0cc770e80052](https://gist.github.com/imdhmd/cda6880e0cc770e80052)) ISSUE: Unable to tokenize character code 160 (which looks like space) using whitespace tokenizer ## Create the following mapping and settings ## Settings contains the match\_phrase analyzer definition: curl -XPOST localhost:9200/newindex -d '{ "mappings": { "newtype": { "properties": { "going": { "type": "string", "analyzer" : "match\_phrase" } } } }, "settings": { "number\_of\_shards": 1, "number\_of\_replicas": 0, "analyzer": { "match\_phrase" : { "type": "custom", "tokenizer": "whitespace", "filter": ["lowercase"] } } } }' ## I have indexed the following document ## IMPORTANT NOTE: The space characters in the below document have character code 160. curl -XPOST localhost:9200/newindex/newtype -d ' { "going": "Link ID ADV Router Age Seq# Checksum Link count" }' ## When i search using the following query i do NOT get any results: ## The query below has normal space, but even if it is char code 160 there is no result curl -XPOST localhost:9200/newindex/newtype/\_search -d '{ "query":{ "query\_string":{ "query":"ADV Router" , "fields" : ["going"] } } }' Can you please suggest a fix to this? Is this a known issue? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![brian\_yoder](https://avatars.discourse-cdn.com/v4/letter/b/f1d935/32.png) [@brian\_yoder](https://discuss.elastic.co/u/brian_yoder)\ **Post date:** [July 7, 2013, 6:21pm UTC](https://discuss.elastic.co/t/whitespace-tokenizer-does-not-tokenise-character-code-160/12675/2 "2013-07-07T18:21:16Z")
Hi, Imdad, One work-around is to define a char filter that converts code 160 to code 20 (ASCII blank) before tokenizing. I am not on my usual laptop and can't give you an example. But I do this for matching various languages in which there are character equivalencies for matching (for example, Finnish tends like like W to match V). Regards, Brian On Friday, July 5, 2013 6:06:15 AM UTC-4, Imdad Ahmed wrote: > Hello All, > > I'm using ES 0.20.6. > > (all the curl commands below are available at this gist: > [https://gist.github.com/imdhmd/cda6880e0cc770e80052](https://gist.github.com/imdhmd/cda6880e0cc770e80052)) > > ISSUE: Unable to tokenize character code 160 (which looks like space) > using whitespace tokenizer > > ## Create the following mapping and settings > > ## Settings contains the match\_phrase analyzer definition: > > curl -XPOST localhost:9200/newindex -d '{ > "mappings": { > "newtype": { > "properties": { > "going": { > "type": "string", "analyzer" : "match\_phrase" > } > } > } > }, > "settings": { > "number\_of\_shards": 1, > "number\_of\_replicas": 0, > "analyzer": { > "match\_phrase" : { > "type": "custom", > "tokenizer": "whitespace", > "filter": ["lowercase"] > } > } > } > }' > > ## I have indexed the following document > > ## IMPORTANT NOTE: The space characters in the below document have > > character code 160. > > curl -XPOST localhost:9200/newindex/newtype -d ' > { > "going": "Link ID ADV Router Age Seq# > Checksum Link count" > }' > > ## When i search using the following query i do NOT get any results: > > ## The query below has normal space, but even if it is char code 160 there > > is no result > > curl -XPOST localhost:9200/newindex/newtype/\_search -d '{ > "query":{ > "query\_string":{ > "query":"ADV Router" > , > "fields" : ["going"] > } > } > }' > > Can you please suggest a fix to this? Is this a known issue? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [July 7, 2013, 11:28pm UTC](https://discuss.elastic.co/t/whitespace-tokenizer-does-not-tokenise-character-code-160/12675/3 "2013-07-07T23:28:09Z")
This is an issue with Lucene. Lucene whitespace tokenizer only checks whitespace for Java, which is realized by Character.isWhiteSpace() [http://docs.oracle.com/javase/6/docs/api/java/lang/Character.html#isWhitespace(char)](http://docs.oracle.com/javase/6/docs/api/java/lang/Character.html#isWhitespace(char)) But the Java whitespaces are unfortunately different from Unicode whitespace property list in [http://www.unicode.org/Public/UCD/latest/ucd/PropList.txt](http://www.unicode.org/Public/UCD/latest/ucd/PropList.txt) Jörg Am 05.07.13 12:06, schrieb Imdad Ahmed: > ISSUE: Unable to tokenize character code 160 (which looks like space) > using whitespace tokenizer -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [July 7, 2013, 11:42pm UTC](https://discuss.elastic.co/t/whitespace-tokenizer-does-not-tokenise-character-code-160/12675/4 "2013-07-07T23:42:31Z")
I opened an improvement issue [https://issues.apache.org/jira/browse/LUCENE-5096](https://issues.apache.org/jira/browse/LUCENE-5096) Jörg Am 08.07.13 01:28, schrieb Jörg Prante: > This is an issue with Lucene. Lucene whitespace tokenizer only checks > whitespace for Java, which is realized by Character.isWhiteSpace() > [Character (Java Platform SE 6)](http://docs.oracle.com/javase/6/docs/api/java/lang/Character.html#isWhitespace(char)) > > But the Java whitespaces are unfortunately different from Unicode > whitespace property list in > [http://www.unicode.org/Public/UCD/latest/ucd/PropList.txt](http://www.unicode.org/Public/UCD/latest/ucd/PropList.txt) > > Jörg > > Am 05.07.13 12:06, schrieb Imdad Ahmed: > > > ISSUE: Unable to tokenize character code 160 (which looks like space) > > using whitespace tokenizer -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![imdhmd](https://avatars.discourse-cdn.com/v4/letter/i/d07c76/32.png) [@imdhmd](https://discuss.elastic.co/u/imdhmd)\ **Post date:** [July 8, 2013, 5:00am UTC](https://discuss.elastic.co/t/whitespace-tokenizer-does-not-tokenise-character-code-160/12675/5 "2013-07-08T05:00:07Z")
Thanks Jörg On Monday, July 8, 2013 5:12:31 AM UTC+5:30, Jörg Prante wrote: > I opened an improvement issue > [[LUCENE-5096] WhitespaceTokenizer supports Java whitespace, should also support Unicode whitespace - ASF JIRA](https://issues.apache.org/jira/browse/LUCENE-5096) > > Jörg > > Am 08.07.13 01:28, schrieb Jörg Prante: > > > This is an issue with Lucene. Lucene whitespace tokenizer only checks > > whitespace for Java, which is realized by Character.isWhiteSpace() > > [Character (Java Platform SE 6)](http://docs.oracle.com/javase/6/docs/api/java/lang/Character.html#isWhitespace(char)) > > > But the Java whitespaces are unfortunately different from Unicode > > whitespace property list in > > [http://www.unicode.org/Public/UCD/latest/ucd/PropList.txt](http://www.unicode.org/Public/UCD/latest/ucd/PropList.txt) > > > > Jörg > > > > Am 05.07.13 12:06, schrieb Imdad Ahmed: > > > > > ISSUE: Unable to tokenize character code 160 (which looks like space) > > > using whitespace tokenizer -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![imdhmd](https://avatars.discourse-cdn.com/v4/letter/i/d07c76/32.png) [@imdhmd](https://discuss.elastic.co/u/imdhmd)\ **Post date:** [July 8, 2013, 8:17am UTC](https://discuss.elastic.co/t/whitespace-tokenizer-does-not-tokenise-character-code-160/12675/6 "2013-07-08T08:17:18Z")
For the benefits of others: char\_filter: whitespace\_mapping: type: mapping mappings: ["\u00A0=\>\u0020"] On Monday, July 8, 2013 10:30:07 AM UTC+5:30, Imdad Ahmed wrote: > Thanks Jörg > > On Monday, July 8, 2013 5:12:31 AM UTC+5:30, Jörg Prante wrote: > > > I opened an improvement issue > > [[LUCENE-5096] WhitespaceTokenizer supports Java whitespace, should also support Unicode whitespace - ASF JIRA](https://issues.apache.org/jira/browse/LUCENE-5096) > > > > Jörg > > > > Am 08.07.13 01:28, schrieb Jörg Prante: > > > > > This is an issue with Lucene. Lucene whitespace tokenizer only checks > > > whitespace for Java, which is realized by Character.isWhiteSpace() > > > > [Character (Java Platform SE 6)](http://docs.oracle.com/javase/6/docs/api/java/lang/Character.html#isWhitespace(char)) > > > > > But the Java whitespaces are unfortunately different from Unicode > > > whitespace property list in > > > [http://www.unicode.org/Public/UCD/latest/ucd/PropList.txt](http://www.unicode.org/Public/UCD/latest/ucd/PropList.txt) > > > > > > Jörg > > > > > > Am 05.07.13 12:06, schrieb Imdad Ahmed: > > > > > > > ISSUE: Unable to tokenize character code 160 (which looks like space) > > > > using whitespace tokenizer -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:27am UTC](https://discuss.elastic.co/t/whitespace-tokenizer-does-not-tokenise-character-code-160/12675/7 "2017-07-06T02:27:42Z")