# Problems with Ascii Folding text with Accents

**URL:** <https://discuss.elastic.co/t/problems-with-ascii-folding-text-with-accents/38307>\
**Category:** Elasticsearch\
**Created:** [January 4, 2016, 11:53am UTC](https://discuss.elastic.co/t/problems-with-ascii-folding-text-with-accents/38307 "2016-01-04T11:53:32Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![dadepo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadepo/32/19944_2.png) [@dadepo](https://discuss.elastic.co/u/dadepo)\
**Post date:** [January 4, 2016, 11:53am UTC](https://discuss.elastic.co/t/problems-with-ascii-folding-text-with-accents/38307/1 "2016-01-04T11:53:32Z")

</div>

I have the following analysis set up:

```
  "asciifolding_analyzer": {
    "char_filter": ["icu_normalizer"],
    "tokenizer": "standard",
    "filter": ["lowercase", "asciifolding"]
  }

```

but when I try it out using the Analyzer API with the following word jọ́jọ́ (notice both the top and bottom accent) i.e.

> GET /testindex/\_analyze?analyzer=asciifolding\_analyzer&text=jọ́jọ́

I get the following results:

```
{
   "tokens": [
      {
         "token": "jójó",
         "start_offset": 0,
         "end_offset": 6,
         "type": "<ALPHANUM>",
         "position": 1
      }
   ]
}

```

Notice that the name was normalized to **jójó** (note the accent on top of the O's). I was expecting it to be normalized to **jojo**. Why is this the case, and any ideas on how to get the ascii-folding to work 'totally'?

---

<div class="post-metadata">

**Author:** ![byronvoorbach](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/byronvoorbach/32/8283_2.png) [@byronvoorbach](https://discuss.elastic.co/u/byronvoorbach)\
**Post date:** [January 4, 2016, 1:01pm UTC](https://discuss.elastic.co/t/problems-with-ascii-folding-text-with-accents/38307/2 "2016-01-04T13:01:30Z")

</div>

Hmm I was able to recreate your issue. Seems like there is an additional normalisation run needed to get 'jojo' as a result:

`localhost:9200/test/_analyze?analyzer=asciifolding_analyzer&text=jọ́jọ́`

gives:

> {"tokens":[{"token":"jójó","start\_offset":0,"end\_offset":6,"type":"","position":1}]}

Analysing again with the result from the previous analysis:

`localhost:9200/test/_analyze?analyzer=asciifolding_analyzer&text=jójó`

gives:

> {"tokens":[{"token":"jojo","start\_offset":0,"end\_offset":4,"type":"","position":1}]}

Not sure why this happens though..

---

<div class="post-metadata">

**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\
**Post date:** [January 4, 2016, 1:25pm UTC](https://discuss.elastic.co/t/problems-with-ascii-folding-text-with-accents/38307/3 "2016-01-04T13:25:17Z")

</div>

`asciifolding` is not aware of Unicode rules to process characters. You should not mix `asciifolding` and ICU.

You should use ICU folding like this

```auto
PUT /test
{
    "settings": {
         "index": {
            "analysis": {
               "analyzer": {
                  "my_icu": {
                     "filter": [
                        "lowercase",
                        "icu_normalizer",
                        "icu_folding"
                     ],
                     "tokenizer": "standard"
                  }
               }
            }
        }
    }
}

```

```auto
POST /test/_analyze?analyzer=my_icu
jọ́jọ́ 
{
  "tokens": [
    {
      "token": "jojo",
      "start_offset": 0,
      "end_offset": 6,
      "type": "<ALPHANUM>",
      "position": 0
    }
  ]
}

```

---

<div class="post-metadata">

**Author:** ![dadepo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadepo/32/19944_2.png) [@dadepo](https://discuss.elastic.co/u/dadepo)\
**Post date:** [January 4, 2016, 2:55pm UTC](https://discuss.elastic.co/t/problems-with-ascii-folding-text-with-accents/38307/4 "2016-01-04T14:55:11Z")

</div>

@jprante Yes indeed. Thanks!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 11:27pm UTC](https://discuss.elastic.co/t/problems-with-ascii-folding-text-with-accents/38307/5 "2017-07-05T23:27:18Z")

</div>