# What kind of encoding does the ES support? **URL:** \ **Category:** Elasticsearch\ **Created:** [March 4, 2014, 3:57am UTC](https://discuss.elastic.co/t/what-kind-of-encoding-does-the-es-support/16127 "2014-03-04T03:57:33Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Ivan\_Ji](https://avatars.discourse-cdn.com/v4/letter/i/d78d45/32.png) [@Ivan\_Ji](https://discuss.elastic.co/u/Ivan_Ji)\ **Post date:** [March 4, 2014, 3:57am UTC](https://discuss.elastic.co/t/what-kind-of-encoding-does-the-es-support/16127/1 "2014-03-04T03:57:33Z")
Hi all, I am wondering about the encoding of ES. What kind of the encoding is of the ES storage? And what's the encoding during the operations? Through the REST API, any json document can be sent to insert into ES. So if I sent an document as follows curl -XPOST '[http://192.168.50.7:9200/data/main/](http://192.168.50.7:9200/data/main/)' -d '{"name":"蒼天", "type":"file", "extension":"tmp", "mime\_type": "application/text"}' As we can see the "name" field is not ascii and assume my native encoding is UTF8. What happened during the insertion? Does it store the original string which is utf8 inside into ES? If the native encoding is not common, ex. Latin, what would happen? Ideas? Regards Ivan -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/07c0ba7b-a5c5-494b-9894-28803c519d34%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/07c0ba7b-a5c5-494b-9894-28803c519d34%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ivan\_Ji](https://avatars.discourse-cdn.com/v4/letter/i/d78d45/32.png) [@Ivan\_Ji](https://discuss.elastic.co/u/Ivan_Ji)\ **Post date:** [March 4, 2014, 6:50am UTC](https://discuss.elastic.co/t/what-kind-of-encoding-does-the-es-support/16127/2 "2014-03-04T06:50:36Z")
Hey all, I just performed several tests. Assume I have the word "愛", its utf8 coding is "\xe6\x84\x9b", and its unicode is '\u611b'. I execute the following command: curl -XPOST '[http://192.168.50.7:9200/data/main](http://192.168.50.7:9200/data/main)" -d '{"name":"愛"}' --\> {"\_index":"data","\_type":"main","\_id":"bFPGRGFaTcqsS1hOfWVjDQ","\_version":1,"created":true} curl -XPOST '[http://192.168.50.7:9200/qusion/main](http://192.168.50.7:9200/qusion/main)' -d '{"name":"\u611b"}' --\> {"\_index":"data","\_type":"main","\_id":"NntYqlwRQl6QZ3ROAclU5w","\_version":1,"created":true} curl -XPOST '[http://192.168.50.7:9200/qusion/main](http://192.168.50.7:9200/qusion/main)' -d '{"name":"\xe6\x84\x9b"}' --\> {"error":"MapperParsingException[failed to parse [name]]; nested: JsonParseException[Unrecognized character escape 'x' (code 120)\n at [Source: [B@b41c580; line: 1, column: 12]]; ","status":400} It seems obvious that json parse exception occur. But what's the difference between '{"name":"\xe6\x84\x9b"}' and '{"name":"愛"}' ? PS. My native lang is utf8. And next I try to find them by using : curl '[http://192.168.50.7:9200/data/main/\_search?pretty](http://192.168.50.7:9200/data/main/_search?pretty)' -d '{"query":{"term":{"name": "愛"}}}' I think it might get only one hit. But it get the two hits. And why? It seem there are some transformation inside it. Ideas? Thanks a lot. Ivan Ji於 2014年3月4日星期二UTC+8上午11時57分33秒寫道： > Hi all, > > I am wondering about the encoding of ES. What kind of the encoding is of > the ES storage? And what's the encoding during the operations? > > Through the REST API, any json document can be sent to insert into ES. So > if I sent an document as follows > > curl -XPOST '[http://192.168.50.7:9200/data/main/](http://192.168.50.7:9200/data/main/)' -d '{"name":"蒼天", > "type":"file", "extension":"tmp", "mime\_type": "application/text"}' > > As we can see the "name" field is not ascii and assume my native encoding > is UTF8. What happened during the insertion? > Does it store the original string which is utf8 inside into ES? If the > native encoding is not common, ex. Latin, what would happen? > > Ideas? > > Regards > > Ivan -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/f0b10c56-0e88-4670-8d1f-a9b6576f1543%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f0b10c56-0e88-4670-8d1f-a9b6576f1543%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ivan\_Ji](https://avatars.discourse-cdn.com/v4/letter/i/d78d45/32.png) [@Ivan\_Ji](https://discuss.elastic.co/u/Ivan_Ji)\ **Post date:** [March 4, 2014, 6:56am UTC](https://discuss.elastic.co/t/what-kind-of-encoding-does-the-es-support/16127/3 "2014-03-04T06:56:15Z")
Hey all, I just performed several tests. Assume I have the word "愛", its utf8 coding is "\xe6\x84\x9b", and its unicode is '\u611b'. I execute the following command: curl -XPOST '[http://192.168.50.7:9200/data/main](http://192.168.50.7:9200/data/main)" -d '{"name":"愛"}' --\> {"\_index":"data","\_type":"main","\_id":"bFPGRGFaTcqsS1hOfWVjDQ","\_version":1,"created":true} curl -XPOST '[http://192.168.50.7:9200/data/main](http://192.168.50.7:9200/data/main)' -d '{"name":"\u611b"}' --\> {"\_index":"data","\_type":"main","\_id":"NntYqlwRQl6QZ3ROAclU5w","\_version":1,"created":true} curl -XPOST '[http://192.168.50.7:9200/data/main](http://192.168.50.7:9200/data/main)' -d '{"name":"\xe6\x84\x9b"}' --\> {"error":"MapperParsingException[failed to parse [name]]; nested: JsonParseException[Unrecognized character escape 'x' (code 120)\n at [Source: [B@b41c580; line: 1, column: 12]]; ","status":400} It seems obvious that json parse exception occur. But what's the difference between '{"name":"\xe6\x84\x9b"}' and '{"name":"愛"}' ? PS. My native lang is utf8. And next I try to find them by using : curl '[http://192.168.50.7:9200/data/main/\_search?pretty](http://192.168.50.7:9200/data/main/_search?pretty)' -d '{"query":{"term":{"name": "愛"}}}' I think it might get only one hit. But it get the two hits. And why? It seem there are some transformation inside it. Ideas? Thanks a lot. Ivan Ji於 2014年3月4日星期二UTC+8上午11時57分33秒寫道： > Hi all, > > I am wondering about the encoding of ES. What kind of the encoding is of > the ES storage? And what's the encoding during the operations? > > Through the REST API, any json document can be sent to insert into ES. So > if I sent an document as follows > > curl -XPOST '[http://192.168.50.7:9200/data/main/](http://192.168.50.7:9200/data/main/)' -d '{"name":"蒼天", > "type":"file", "extension":"tmp", "mime\_type": "application/text"}' > > As we can see the "name" field is not ascii and assume my native encoding > is UTF8. What happened during the insertion? > Does it store the original string which is utf8 inside into ES? If the > native encoding is not common, ex. Latin, what would happen? > > Ideas? > > Regards > > Ivan -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/593a6fbe-7e37-4f82-b706-f92bc0785d41%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/593a6fbe-7e37-4f82-b706-f92bc0785d41%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![brian\_yoder](https://avatars.discourse-cdn.com/v4/letter/b/f1d935/32.png) [@brian\_yoder](https://discuss.elastic.co/u/brian_yoder)\ **Post date:** [March 4, 2014, 11:37pm UTC](https://discuss.elastic.co/t/what-kind-of-encoding-does-the-es-support/16127/4 "2014-03-04T23:37:14Z")
Ivan, Yes, ES stores all strings in UTF-8 encoding. Referring to your 3 POST commands, the first two succeeded because in the first one, you presented the data in the UTF-8 encoding and it was accepted. In the second one, you presented the same name but in using the \u notation which is valid, supported, and equivalent in value. And since you created the documents and let ES automatically assign the id, you have 2 spearate documents, and both have the same name. Hence your final query matches them both. No mystery there. Your third command failed because the \x notation is not valid JSON. Looking at [http://www.json.org/](http://www.json.org/) it can be seen that the JSON specification accepts \u notation but not the \x notation. And from the error message, the (most excellent) Jackson library follows the specification closely on this. Hope this helps! Brian -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/fa1fc5dc-3f52-4863-92e5-ec06f6abcadc%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/fa1fc5dc-3f52-4863-92e5-ec06f6abcadc%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:45am UTC](https://discuss.elastic.co/t/what-kind-of-encoding-does-the-es-support/16127/5 "2017-07-06T01:45:43Z")