# Dealing with diacritic values (Example :- Lehtelä)

**URL:** <https://discuss.elastic.co/t/dealing-with-diacritic-values-example-lehtela/202987>\
**Category:** Elasticsearch\
**Created:** [October 10, 2019, 9:12am UTC](https://discuss.elastic.co/t/dealing-with-diacritic-values-example-lehtela/202987 "2019-10-10T09:12:13Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![MKU](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mku/32/48327_2.png) [@MKU](https://discuss.elastic.co/u/MKU)\
**Post date:** [October 10, 2019, 9:12am UTC](https://discuss.elastic.co/t/dealing-with-diacritic-values-example-lehtela/202987/1 "2019-10-10T09:12:13Z")

</div>

Hi all,  
I have some values which has special symbols which helps sounding of the word.  
The problem is these values are getting encoded to **base64** while indexing, for example above term is indexed as **TGVodGVsw6Q=** , and if my query is **vod** its retrieving the document with TGVodGVsw6Q , which is wrong.  
How do i address this situation.  
I want the value to be stored as it is (Lehtelä) and still be searchable.  
By the way i also tried this filter

```auto
"filter": {
"my_ascii_folding": {
"type": "asciifolding",
"preserve_original": "true"
}

```

and it did not work  
Thanks

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [October 11, 2019, 12:32pm UTC](https://discuss.elastic.co/t/dealing-with-diacritic-values-example-lehtela/202987/2 "2019-10-11T12:32:16Z")

</div>

Hey,

I'm sorry, but I do not fully understand the issue here, as it seems to me, that two problems are intermingled here. Please correct me, where I am wrong

1. Content seems to be in base64. This is superbad and should be fixed. You can decode this using an ingest processor. See this example

```auto
POST /_ingest/pipeline/_simulate
{
  "pipeline" :
  {
    "description": "_description",
    "processors": [
      {
        "script": {
          "lang": "painless",
          "source": "ctx.decoded = ctx.field.decodeBase64()"
        }
      }
    ]
  },
  "docs": [
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "field" : "TGVodGVsw6Q="
      }
    }
  ]
}

```

1. Ascii folding is not working as expected

```auto
GET _analyze
{
  "text": "Lehtelä",
  "filter": [{ "type" : "asciifolding", "preserve_original": "true"}]
}

{
  "tokens" : [
    {
      "token" : "Lehtela",
      "start_offset" : 0,
      "end_offset" : 7,
      "type" : "word",
      "position" : 0
    },
    {
      "token" : "Lehtelä",
      "start_offset" : 0,
      "end_offset" : 7,
      "type" : "word",
      "position" : 0
    }
  ]
}

```

that looks as expected to me.

Happy to see some more clarification on this. What you cannot do, is just storing the base64 value and expect everything to work magically. You need to do some conversion.

--Alex

---

<div class="post-metadata">

**Author:** ![MKU](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mku/32/48327_2.png) [@MKU](https://discuss.elastic.co/u/MKU)\
**Post date:** [October 14, 2019, 4:52am UTC](https://discuss.elastic.co/t/dealing-with-diacritic-values-example-lehtela/202987/3 "2019-10-14T04:52:33Z")

</div>

Hi,  
Thanks for your reply.  
I was able to figure out the problem, The data for elasticsearch i took it from an LDAP server. LDAP was actually encoding those non ASCII values.  
I think i should fix it from LDAP side.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 11, 2019, 4:52am UTC](https://discuss.elastic.co/t/dealing-with-diacritic-values-example-lehtela/202987/4 "2019-11-11T04:52:34Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.