# Icu\_collation as keyword normalizer

**URL:** <https://discuss.elastic.co/t/icu-collation-as-keyword-normalizer/74688>\
**Category:** Elasticsearch\
**Created:** [February 10, 2017, 4:53pm UTC](https://discuss.elastic.co/t/icu-collation-as-keyword-normalizer/74688 "2017-02-10T16:53:44Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![mabn0](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mabn0/32/15135_2.png) [@mabn0](https://discuss.elastic.co/u/mabn0)\
**Post date:** [February 10, 2017, 4:53pm UTC](https://discuss.elastic.co/t/icu-collation-as-keyword-normalizer/74688/1 "2017-02-10T16:53:44Z")

</div>

I'd like to use icu\_collation to normalize fields with "keyword" type - the goal is to make sorting behave properly on string in a certain language. But ElasticSearch does not let me do that - it complains that: **"Custom normalizer [my\_normalizer] may not use filter [icu\_collation]"**

For this mapping:

```
PUT /sort_test/
{
 "settings": {
    "analysis": {
      "normalizer": {
        "my_normalizer": {
          "type": "custom",
          "filter" : ["icu_collation"]
        }
      }
    }
  },
  "mappings": {
    "my_type": {
      "properties": {
        "word": {
          "type": "keyword",
          "normalizer" : "my_normalizer"
        }
      }
    }
  }
}

```

After replacing icu\_collation with icu\_normalizer the mapping is accepted by ElasticSearch, but I need sorting, not normalization.  
Is it possible to use icu\_collation with keywords (or in general without field data)?

---

<div class="post-metadata">

**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\
**Post date:** [February 10, 2017, 7:15pm UTC](https://discuss.elastic.co/t/icu-collation-as-keyword-normalizer/74688/2 "2017-02-10T19:15:59Z")

</div>

I recommend my implementation of ICU collation key analyzer

> <https://github.com/jprante/elasticsearch-analysis-icu/blob/master/src/main/java/org/xbib/elasticsearch/index/analysis/icu/IcuCollationKeyAnalyzer.java>

Example

```auto
PUT /sort_test/
{
 "settings": {
    "analysis": {
      "analyzer": {
        "my_collator": {
          "type": "icu_collation",
          "language" : "de",
          "country" : "DE",
          "strength" : "primary",
          "rules" : "& ae , a\u0308 & AE , A\u0308& oe , o\u0308 & OE , O\u0308& ue , u\u0308 & UE , u\u0308"  
        }
      }
    }
  },
  "mappings": {
    "my_type": {
      "properties": {
        "word": {
          "type": "text",
          "analyzer" : "my_collator",
          "store": true
        }
      }
    }
  }
}

```

My ICU collation key analyzer creates byte sequences which can be used as a sort key in ES. With `store:true` it should be possible to sort on field `word`.

---

<div class="post-metadata">

**Author:** ![mabn0](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mabn0/32/15135_2.png) [@mabn0](https://discuss.elastic.co/u/mabn0)\
**Post date:** [February 13, 2017, 1:56pm UTC](https://discuss.elastic.co/t/icu-collation-as-keyword-normalizer/74688/3 "2017-02-13T13:56:14Z")

</div>

Even with `store=true` it complains about disabled field data for text fields.

I think it will be easier to just use ICU in the application and send to elasticsearch documents with already computed collation key (inside keyword field).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 13, 2017, 1:56pm UTC](https://discuss.elastic.co/t/icu-collation-as-keyword-normalizer/74688/4 "2017-03-13T13:56:14Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.