# What does the rbbi rule for icu\_tokenizer in the docs do?

**URL:** <https://discuss.elastic.co/t/what-does-the-rbbi-rule-for-icu-tokenizer-in-the-docs-do/121026>\
**Category:** Elasticsearch\
**Created:** [February 22, 2018, 10:25am UTC](https://discuss.elastic.co/t/what-does-the-rbbi-rule-for-icu-tokenizer-in-the-docs-do/121026 "2018-02-22T10:25:24Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Judy\_Raj](https://avatars.discourse-cdn.com/v4/letter/j/ccd318/32.png) [@Judy\_Raj](https://discuss.elastic.co/u/Judy_Raj)\
**Post date:** [February 22, 2018, 10:25am UTC](https://discuss.elastic.co/t/what-does-the-rbbi-rule-for-icu-tokenizer-in-the-docs-do/121026/1 "2018-02-22T10:25:24Z")

</div>

I was trying to work with the icu tokenizer and came across the rbbi rules section in the docs.  
[https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-icu-tokenizer.html](https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-icu-tokenizer.html)  
I used the same rule from the docs .+ {200}; for Arabic but my result looks nothing like the one in the docs.  
Could someone perhaps explain what does this .+ {200}; mean?

---

<div class="post-metadata">

**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\
**Post date:** [February 22, 2018, 7:28pm UTC](https://discuss.elastic.co/t/what-does-the-rbbi-rule-for-icu-tokenizer-in-the-docs-do/121026/2 "2018-02-22T19:28:16Z")

</div>

Unicode has a concept of scripts, e.g. "Latn", "Cyrl", "Arab" etc. See ISO 15924 and [http://www.unicode.org/reports/tr24/](http://www.unicode.org/reports/tr24/)

The ICU tokenizer can be configured to use a custom break iterator for segmentation of words. One type of break iterator is a rule-based break iterator (RBBI). The set of rules are typically for a script. Thus, loading of RBBI rules are specified like "Latn:name.rbbi".

More information about ICU rules for finding boundary positions within text can be found at [http://userguide.icu-project.org/boundaryanalysis#TOC-RBBI-Rules](http://userguide.icu-project.org/boundaryanalysis#TOC-RBBI-Rules)

So, as a very basic example, the KeywordTokenizer.rbbi

> <https://github.com/apache/lucene-solr/blob/master/lucene/analysis/icu/src/test/org/apache/lucene/analysis/icu/segmentation/KeywordTokenizer.rbbi>

consists of the single rule ".+ {200}", which means, "take latin script, take one character, no matter what character, and repeat for all characters in the input until there is no more left, and map the result to token class 200 (which stands for "alphanumerical tokens")".

This rule effectively ignores word boundaries and takes the whole input as a single token (which is equivalent to the Lucene KeywordTokenizer).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 22, 2018, 7:28pm UTC](https://discuss.elastic.co/t/what-does-the-rbbi-rule-for-icu-tokenizer-in-the-docs-do/121026/3 "2018-03-22T19:28:40Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.