# Convert existing character encoding in ElasticSearch

**URL:** https://discuss.elastic.co/t/convert-existing-character-encoding-in-elasticsearch/182240
**Category:** Elasticsearch
**Created:** [May 22, 2019, 1:48pm UTC](https://discuss.elastic.co/t/convert-existing-character-encoding-in-elasticsearch/182240 "2019-05-22T13:48:21Z")
**Posts on this page:** 2
**Page:** 1

<div class="post-metadata">

### Author: ![markmevans](https://avatars.discourse-cdn.com/v4/letter/m/22d042/32.png) [@markmevans](https://discuss.elastic.co/u/markmevans)
#### Post date: [May 22, 2019, 1:48pm UTC](https://discuss.elastic.co/t/convert-existing-character-encoding-in-elasticsearch/182240/1 "2019-05-22T13:48:21Z")

</div>

The was ElasticSearch received JSON from Java was amended from ISO-8891-1 to UTF-8 because we were getting invalid JSON when symbols such as the copyright sign were at the start of a field.

However, I now notice that since this change characters such as é are being stored in ElasticSearch as Ã©

This is because we were still amending the encoding of the string prior to ElasticSearch insert as follows:

```
                    byte bytes[] = newString.getBytes("UTF-8");
		newString = new String(bytes, "ISO-8859-1");

```

When I take out this string formatting it now once again inserts correctly into ElasticSearch.

However, we have tens of thousands of records which have been inserted with words such as "naÃ¯ve" instead of "naïve" etc. - I was wondering if there is any way of converting these back to their UTF-8 equivalent?

Taking the character conversion from [https://www.i18nqa.com/debug/utf8-debug.html](https://www.i18nqa.com/debug/utf8-debug.html) - I have tried Logstash to read from one index and write to a new one with a filter as follows:  
filter {  
mutate {  
gsub =\> [  
# replace backslashes, question marks, hashes, and minuses  
# with a dot "."  
"ARTICLE\_TITLE", "Ã¯", "ï",  
"ARTICLE\_TITLE", "Ã©", "é",  
"ARTICLE\_TITLE", "Ãº", "ú",  
"ARTICLE\_TITLE", "Å¯", "ů",  
"ARTICLE\_TITLE", "Ã­", "í",  
"ARTICLE\_TITLE", "Ã¡", "á",  
"ARTICLE\_TITLE", "Å", "ř",  
"ARTICLE\_TITLE", "Ã", "×",  
"ARTICLE\_TITLE", "Ã¦", "æ",  
"ARTICLE\_TITLE", "Ã³", "ó"  
]

But there are many, many more example characters and a good chance some may be missed using a one to one character mapping - I was wondering if there is a more efficient way of converting all the characters interpreted as Windows-1252 (or ISO 8859-1) bytes to UTF-8 bytes in either logstash or by running an ElasticSearch update?

Many thanks,  
Mark

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 19, 2019, 1:48pm UTC](https://discuss.elastic.co/t/convert-existing-character-encoding-in-elasticsearch/182240/2 "2019-06-19T13:48:28Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
