# How does the ES to handle strings with different encodings?

**URL:** <https://discuss.elastic.co/t/how-does-the-es-to-handle-strings-with-different-encodings/16100>\
**Category:** Elasticsearch\
**Created:** [March 3, 2014, 9:11am UTC](https://discuss.elastic.co/t/how-does-the-es-to-handle-strings-with-different-encodings/16100 "2014-03-03T09:11:56Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ivan\_Ji](https://avatars.discourse-cdn.com/v4/letter/i/d78d45/32.png) [@Ivan\_Ji](https://discuss.elastic.co/u/Ivan_Ji)\
**Post date:** [March 3, 2014, 9:11am UTC](https://discuss.elastic.co/t/how-does-the-es-to-handle-strings-with-different-encodings/16100/1 "2014-03-03T09:11:56Z")

</div>

Hi all,

I am using python to communicate with ES. I know the ES use UTF8 encoding.

But when I insert a unicode into ES, does it try to encode it into UTF8?

Because I encounter an situation, I insert two documents into ES. The first  
document contains a unicode string ex. u'\u611b'.  
And the second one has the _same_ _content_ of the first one, but the  
encoding of the string is utf8, ex. '\xe6\x84\x9b'.

And no matter what encoding of string that I use in search, I can both get  
the _two_ _hits_. It seems excellent but I am wondering why?  
Does ES normalize the encoding when indexing and querying? Or how can it  
find the results with different encoding?

Any idea?

Ivan

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/dae9f8c9-da23-4752-8a0b-18bda2cbefbd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/dae9f8c9-da23-4752-8a0b-18bda2cbefbd%40googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Honza\_Kral](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/honza_kral/32/876_2.png) [@Honza\_Kral](https://discuss.elastic.co/u/Honza_Kral)\
**Post date:** [March 3, 2014, 12:34pm UTC](https://discuss.elastic.co/t/how-does-the-es-to-handle-strings-with-different-encodings/16100/2 "2014-03-03T12:34:20Z")

</div>

Hi Ivan,

the python client should accept both strings and unicode though I  
highly recommend you always use unicode to avoid encoding issues. When  
it is sent to the server we always encode the data in UTF-8 so both  
these strings will end up encoded the same.

To see exactly what's going on you can simple enable logging of  
individual requests to see the json being sent to the server, to do  
that just do:

```
import logging
tracer = logging.getLogger('elasticsearch.trace')
tracer.setLevel(logging.INFO)
tracer.addHandler(logging.FileHandler('/tmp/es_trace.log'))

```

then in /tmp/es\_trace.log you will have a transcript of your session  
as curl commands.

Hope this helps,  
Honza

On Mon, Mar 3, 2014 at 10:11 AM, Ivan Ji [hxuanji@gmail.com](mailto:hxuanji@gmail.com) wrote:

> Hi all,
> 
> I am using python to communicate with ES. I know the ES use UTF8 encoding.
> 
> But when I insert a unicode into ES, does it try to encode it into UTF8?
> 
> Because I encounter an situation, I insert two documents into ES. The first  
> document contains a unicode string ex. u'\u611b'.  
> And the second one has the same content of the first one, but the encoding  
> of the string is utf8, ex. '\xe6\x84\x9b'.
> 
> And no matter what encoding of string that I use in search, I can both get  
> the two hits. It seems excellent but I am wondering why?  
> Does ES normalize the encoding when indexing and querying? Or how can it  
> find the results with different encoding?
> 
> Any idea?
> 
> Ivan
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/dae9f8c9-da23-4752-8a0b-18bda2cbefbd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/dae9f8c9-da23-4752-8a0b-18bda2cbefbd%40googlegroups.com).  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CABfdDirUVmOoXaFMpcxXCegjaYtfyy2wX%2BrEL9hCG\_%3DeqSrFPQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CABfdDirUVmOoXaFMpcxXCegjaYtfyy2wX%2BrEL9hCG_%3DeqSrFPQ%40mail.gmail.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:46am UTC](https://discuss.elastic.co/t/how-does-the-es-to-handle-strings-with-different-encodings/16100/3 "2017-07-06T01:46:12Z")

</div>