# Json Binary Format **URL:** \ **Category:** Elasticsearch\ **Created:** [September 15, 2010, 11:00am UTC](https://discuss.elastic.co/t/json-binary-format/3338 "2010-09-15T11:00:21Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [September 15, 2010, 11:00am UTC](https://discuss.elastic.co/t/json-binary-format/3338/1 "2010-09-15T11:00:21Z")
Hi, ``` elasticsearch has support a binary format similar to json for some time ``` now, named "xson". This was used internally when using the Java API when possible (for example, a search request used it automatically). The good news is that the good people at Jackson (actually Tatu) has created an excellent binary format for JSON called Smile. I have already pushed support for that, moreover, I have removed XSON completely and now use Smile as the formal binary format for JSON. Another good news is that the Smile format is public and properly documented. You can find it here: [http://wiki.fasterxml.com/JacksonBinaryFormat](http://wiki.fasterxml.com/JacksonBinaryFormat), with the format here: [http://wiki.fasterxml.com/JacksonBinaryFormatSpec](http://wiki.fasterxml.com/JacksonBinaryFormatSpec). I must say that this is one of the best binary formats I have seen for JSON (BSON is an abomination). It is a bit complicated in some places (zigzagging numbers, and so on), but well worth the effort. I am posting it here since if someone from other lang camps here can maybe take it and implement it in the respective language. I am trying to work with Tatu to get a proper web site to it and to make it feel more formal. This _should_ be the formal binary representation of JSON. As far as elasticsearch, it plays with both easily, on all layers (including REST). Moreover, you can push data in one format and get it in another format easily (for example, JSON for debugging, but Smile as the default). With the REST layer, you don't have to provide content type, as its detected automatically based on the body, or you can simply sent a content type of application/smile for API level, source level format is automatically detected. Also, the response is automatically send using the format of the request. On the Java end, there has been some refacoring which should not affect most users. If you were using the XContentBuilder base class, then nothing changes. The BinaryXContentBuilder and StringXContentBuilder were removed and now we only have XContentBuilder. The XSON format has basically been renamed to Smile format. As a side note, elasticsearch has an abstraction on top of its both parsing and building of "xcontent", so, for example, support other fomrats (like XML) should not be too difficult to add. Are you interested in it? -shay.banon ---
**Author:** ![medcl\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/medcl_2/32/3297_2.png) [@medcl\_2](https://discuss.elastic.co/u/medcl_2)\ **Post date:** [October 5, 2010, 12:27pm UTC](https://discuss.elastic.co/t/json-binary-format/3338/2 "2010-10-05T12:27:55Z")
hey,shay.banon ，i am using thrift in .net,and the type of request's body and response's body is byte array，Method with Get is fine，and the post method returned the following error，and i am a little confused about xcontent and smile， i think i should use the right type to serialize request.body or deserialize the response.body,so what can i do？: [2010-10-05 20:05:39,999][DEBUG][action.index] [Abominatrix] [index ][2], node[7cfee996-5fc0-4048-8137-b812ac0c7c5f], [P], s[STARTED]: Failed to exe cute [index {[index][type][bc40f94b-aba4-43c8-a7d3-26517140ccd0], source[? 1, 34, 125, 0, 0] at org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFa ctory.java:132) at org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFa ctory.java:123) at org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse( XContentDocumentMapper.java:321) at org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse( XContentDocumentMapper.java:308) at org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreat e(InternalIndexShard.java:208) at org.elasticsearch.action.index.TransportIndexAction.shardOperationOnP rimary(TransportIndexAction.java:140) at org.elasticsearch.action.index.TransportIndexAction.shardOperationOnP rimary(TransportIndexAction.java:62) at org.elasticsearch.action.support.replication.TransportShardReplicatio nOperationAction $AsyncShardOperationAction.performOnPrimary(TransportShardReplic ationOperationAction.java:385) at org.elasticsearch.action.support.replication.TransportShardReplicatio nOperationAction$AsyncShardOperationAction.access $400(TransportShardReplicationO perationAction.java:212) at org.elasticsearch.action.support.replication.TransportShardReplicatio nOperationAction$AsyncShardOperationAction $1.run(TransportShardReplicationOperat ionAction.java:282) at java.util.concurrent.ThreadPoolExecutor $Worker.runTask(ThreadPoolExec utor.java:886) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor .java:908) at java.lang.Thread.run(Thread.java:619) On 9月15日, 下午7时00分, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > Hi, > > ``` > elasticsearch has support a binary format similar to json for some time > > ``` > > now, named "xson". This was used internally when using the Java API when > possible (for example, a search request used it automatically). The good > news is that the good people at Jackson (actually Tatu) has created an > excellent binary format for JSON called Smile. I have already pushed support > for that, moreover, I have removed XSON completely and now use Smile as the > formal binary format for JSON. > > Another good news is that the Smile format is public and properly > documented. You can find it here:[http://wiki.fasterxml.com/JacksonBinaryFormat](http://wiki.fasterxml.com/JacksonBinaryFormat), with the format here:[http://wiki.fasterxml.com/JacksonBinaryFormatSpec](http://wiki.fasterxml.com/JacksonBinaryFormatSpec). I must say that this is > one of the best binary formats I have seen for JSON (BSON is an > abomination). It is a bit complicated in some places (zigzagging numbers, > and so on), but well worth the effort. > > I am posting it here since if someone from other lang camps here can > maybe take it and implement it in the respective language. I am trying to > work with Tatu to get a proper web site to it and to make it feel more > formal. This _should_ be the formal binary representation of JSON. > > As far as elasticsearch, it plays with both easily, on all layers > (including REST). Moreover, you can push data in one format and get it in > another format easily (for example, JSON for debugging, but Smile as the > default). With the REST layer, you don't have to provide content type, as > its detected automatically based on the body, or you can simply sent a > content type of application/smile for API level, source level format is > automatically detected. Also, the response is automatically send using the > format of the request. > > On the Java end, there has been some refacoring which should not affect > most users. If you were using the XContentBuilder base class, then nothing > changes. The BinaryXContentBuilder and StringXContentBuilder were removed > and now we only have XContentBuilder. The XSON format has basically been > renamed to Smile format. > > As a side note, elasticsearch has an abstraction on top of its both > parsing and building of "xcontent", so, for example, support other fomrats > (like XML) should not be too difficult to add. Are you interested in it? > > -shay.banon ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [October 5, 2010, 12:30pm UTC](https://discuss.elastic.co/t/json-binary-format/3338/3 "2010-10-05T12:30:16Z")
Are you indexing a json? do you convert the json string into UTF8 byte array? 2010/10/5 Medcl [medcl@163.com](mailto:medcl@163.com) > hey,shay.banon ，i am using thrift in .net,and the type of request's > body and response's body is byte array，Method with Get is fine，and > the post method returned the following error，and i am a little > confused about xcontent and smile， i think i should use the right type > to serialize request.body or deserialize the response.body,so what > can i do？: > > [2010-10-05 20:05:39,999][DEBUG][action.index] > [Abominatrix] [index > ][2], node[7cfee996-5fc0-4048-8137-b812ac0c7c5f], [P], s[STARTED]: > Failed to exe > cute [index {[index][type][bc40f94b-aba4-43c8-a7d3-26517140ccd0], > source[? > > 1, 34, 125, 0, 0] > at > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFa > ctory.java:132) > at > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFa > ctory.java:123) > at > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse( > XContentDocumentMapper.java:321) > at > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse( > XContentDocumentMapper.java:308) > at > org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreat > e(InternalIndexShard.java:208) > at > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnP > rimary(TransportIndexAction.java:140) > at > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnP > rimary(TransportIndexAction.java:62) > at > org.elasticsearch.action.support.replication.TransportShardReplicatio > nOperationAction > $AsyncShardOperationAction.performOnPrimary(TransportShardReplic > ationOperationAction.java:385) > at > org.elasticsearch.action.support.replication.TransportShardReplicatio > nOperationAction$AsyncShardOperationAction.access > $400(TransportShardReplicationO > perationAction.java:212) > at > org.elasticsearch.action.support.replication.TransportShardReplicatio > nOperationAction$AsyncShardOperationAction > $1.run(TransportShardReplicationOperat > ionAction.java:282) > at java.util.concurrent.ThreadPoolExecutor > $Worker.runTask(ThreadPoolExec > utor.java:886) > at java.util.concurrent.ThreadPoolExecutor > $Worker.run(ThreadPoolExecutor > .java:908) > at java.lang.Thread.run(Thread.java:619) > On 9月15日, 下午7时00分, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > Hi, > > > > ``` > > elasticsearch has support a binary format similar to json for some > > > > ``` > > time > > > now, named "xson". This was used internally when using the Java API when > > possible (for example, a search request used it automatically). The good > > news is that the good people at Jackson (actually Tatu) has created an > > excellent binary format for JSON called Smile. I have already pushed > > support > > for that, moreover, I have removed XSON completely and now use Smile as > > the > > formal binary format for JSON. > > > > Another good news is that the Smile format is public and properly > > documented. You can find it here: > > [http://wiki.fasterxml.com/JacksonBinaryFormat](http://wiki.fasterxml.com/JacksonBinaryFormat), with the format here: > > [http://wiki.fasterxml.com/JacksonBinaryFormatSpec](http://wiki.fasterxml.com/JacksonBinaryFormatSpec). I must say that this is > > one of the best binary formats I have seen for JSON (BSON is an > > abomination). It is a bit complicated in some places (zigzagging numbers, > > and so on), but well worth the effort. > > > > I am posting it here since if someone from other lang camps here can > > maybe take it and implement it in the respective language. I am trying to > > work with Tatu to get a proper web site to it and to make it feel more > > formal. This _should_ be the formal binary representation of JSON. > > > > As far as elasticsearch, it plays with both easily, on all layers > > (including REST). Moreover, you can push data in one format and get it in > > another format easily (for example, JSON for debugging, but Smile as the > > default). With the REST layer, you don't have to provide content type, as > > its detected automatically based on the body, or you can simply sent a > > content type of application/smile for API level, source level format is > > automatically detected. Also, the response is automatically send using > > the > > format of the request. > > > > On the Java end, there has been some refacoring which should not > > affect > > most users. If you were using the XContentBuilder base class, then > > nothing > > changes. The BinaryXContentBuilder and StringXContentBuilder were removed > > and now we only have XContentBuilder. The XSON format has basically been > > renamed to Smile format. > > > > As a side note, elasticsearch has an abstraction on top of its both > > parsing and building of "xcontent", so, for example, support other > > fomrats > > (like XML) should not be too difficult to add. Are you interested in it? > > > > -shay.banon ---
**Author:** ![medcl\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/medcl_2/32/3297_2.png) [@medcl\_2](https://discuss.elastic.co/u/medcl_2)\ **Post date:** [October 5, 2010, 12:48pm UTC](https://discuss.elastic.co/t/json-binary-format/3338/4 "2010-10-05T12:48:35Z")
yes，and here is the code： TSocket tsocket=new TSocket("localhost",9500); var transport = new TFramedTransport(tsocket); var protocol = new TBinaryProtocol(transport); var client = new Rest.Client(protocol); transport.Open(); var restRequest = new RestRequest(); restRequest.Method = Method.POST; restRequest.Uri = "/index/type"; restRequest.Body=System.Text.UTF8Encoding.UTF8.GetBytes("{"name": "value"}"); ``` restRequest.Parameters=new Dictionary(); restRequest.Parameters.Add("pretty", "true"); var response= client.execute(restRequest); var str= System.Text.UTF8Encoding.Default.GetString(response.Body); ``` On 10月5日, 下午8时30分, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > Are you indexing a json? do you convert the json string into UTF8 byte > array? > > 2010/10/5 Medcl [me...@163.com](mailto:me...@163.com) > > > hey,shay.banon ，i am using thrift in .net,and the type of request's > > body and response's body is byte array，Method with Get is fine，and > > the post method returned the following error，and i am a little > > confused about xcontent and smile， i think i should use the right type > > to serialize request.body or deserialize the response.body,so what > > can i do？: > > > [2010-10-05 20:05:39,999][DEBUG][action.index] > > [Abominatrix] [index > > ][2], node[7cfee996-5fc0-4048-8137-b812ac0c7c5f], [P], s[STARTED]: > > Failed to exe > > cute [index {[index][type][bc40f94b-aba4-43c8-a7d3-26517140ccd0], > > source[? > > > 1, 34, 125, 0, 0] > > at > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFa > > ctory.java:132) > > at > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFa > > ctory.java:123) > > at > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse( > > XContentDocumentMapper.java:321) > > at > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse( > > XContentDocumentMapper.java:308) > > at > > org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreat > > e(InternalIndexShard.java:208) > > at > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnP > > rimary(TransportIndexAction.java:140) > > at > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnP > > rimary(TransportIndexAction.java:62) > > at > > org.elasticsearch.action.support.replication.TransportShardReplicatio > > nOperationAction > > $AsyncShardOperationAction.performOnPrimary(TransportShardReplic > > ationOperationAction.java:385) > > at > > org.elasticsearch.action.support.replication.TransportShardReplicatio > > nOperationAction$AsyncShardOperationAction.access > > $400(TransportShardReplicationO > > perationAction.java:212) > > at > > org.elasticsearch.action.support.replication.TransportShardReplicatio > > nOperationAction$AsyncShardOperationAction > > $1.run(TransportShardReplicationOperat > > ionAction.java:282) > > at java.util.concurrent.ThreadPoolExecutor > > $Worker.runTask(ThreadPoolExec > > utor.java:886) > > at java.util.concurrent.ThreadPoolExecutor > > $Worker.run(ThreadPoolExecutor > > .java:908) > > at java.lang.Thread.run(Thread.java:619) > > On 9月15日, 下午7时00分, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > > > Hi, > > > > ``` > > > elasticsearch has support a binary format similar to json for some > > > > > > ``` > > > > time > > > > > now, named "xson". This was used internally when using the Java API when > > > possible (for example, a search request used it automatically). The good > > > news is that the good people at Jackson (actually Tatu) has created an > > > excellent binary format for JSON called Smile. I have already pushed > > > support > > > for that, moreover, I have removed XSON completely and now use Smile as > > > the > > > formal binary format for JSON. > > > > Another good news is that the Smile format is public and properly > > > documented. You can find it here: > > > [http://wiki.fasterxml.com/JacksonBinaryFormat](http://wiki.fasterxml.com/JacksonBinaryFormat), with the format here: > > > [http://wiki.fasterxml.com/JacksonBinaryFormatSpec](http://wiki.fasterxml.com/JacksonBinaryFormatSpec). I must say that this is > > > one of the best binary formats I have seen for JSON (BSON is an > > > abomination). It is a bit complicated in some places (zigzagging numbers, > > > and so on), but well worth the effort. > > > > I am posting it here since if someone from other lang camps here can > > > maybe take it and implement it in the respective language. I am trying to > > > work with Tatu to get a proper web site to it and to make it feel more > > > formal. This _should_ be the formal binary representation of JSON. > > > > As far as elasticsearch, it plays with both easily, on all layers > > > (including REST). Moreover, you can push data in one format and get it in > > > another format easily (for example, JSON for debugging, but Smile as the > > > default). With the REST layer, you don't have to provide content type, as > > > its detected automatically based on the body, or you can simply sent a > > > content type of application/smile for API level, source level format is > > > automatically detected. Also, the response is automatically send using > > > the > > > format of the request. > > > > On the Java end, there has been some refacoring which should not > > > affect > > > most users. If you were using the XContentBuilder base class, then > > > nothing > > > changes. The BinaryXContentBuilder and StringXContentBuilder were removed > > > and now we only have XContentBuilder. The XSON format has basically been > > > renamed to Smile format. > > > > As a side note, elasticsearch has an abstraction on top of its both > > > parsing and building of "xcontent", so, for example, support other > > > fomrats > > > (like XML) should not be too difficult to add. Are you interested in it? > > > > -shay.banon ---
**Author:** ![medcl\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/medcl_2/32/3297_2.png) [@medcl\_2](https://discuss.elastic.co/u/medcl_2)\ **Post date:** [October 5, 2010, 1:04pm UTC](https://discuss.elastic.co/t/json-binary-format/3338/5 "2010-10-05T13:04:44Z")
logs： [2010-10-05 20:59:10,094][DEBUG][action.index] [Abominatrix] [index][1], node[7cfee996-5fc0-4048-8137-b812ac0c7c5f], [P], s[STARTED]: Failed to execute [index {[index][type][821ce2ad- e20f-4d59-ba77-f75c820dd94e], source[? execute]}] org.elasticsearch.ElasticSearchParseException: Failed to derive xcontent from [-128, 1, 0, 1, 0, 0, 0, 7, 101, 120, 101, 99, 117, 116, 101, 0, 0, 0, 0, 12, 0, 1, 8, 0, 1, 0, 0, 0, 2, 11, 0, 2, 0, 0, 0, 11, 47, 105, 110, 100, 101, 120, 47, 116, 121, 112, 101, 13, 0, 3, 11, 11, 0, 0, 0, 1, 0, 0, 0, 6, 112, 114, 101, 116, 116, 121, 0, 0, 0, 4, 116, 114, 117, 101, 13, 0, 4, 11, 11, 0, 0, 0, 1, 0, 0, 0, 12, 67, 111, 110, 116, 101, 110, 116, 45, 84, 121, 112, 101, 0, 0, 0, 16, 97, 112, 112, 108, 105, 99, 97, 116, 105, 111, 110, 47, 120, 115, 111, 110, 11, 0, 5, 0, 0, 0, 17, 123, 34, 110, 97, 109, 101, 34, 58, 32, 34, 118, 97, 108, 117, 101, 34, 125, 0, 0] at org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory.java: 132) at org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory.java: 123) at org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XContentDocumentMapper.java: 321) at org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XContentDocumentMapper.java: 308) at org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java: 208) at org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java: 140) at org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java: 62) at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction $AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java: 385) at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction $AsyncShardOperationAction.access $400(TransportShardReplicationOperationAction.java:212) at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction $AsyncShardOperationAction $1.run(TransportShardReplicationOperationAction.java:282) at java.util.concurrent.ThreadPoolExecutor $Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) [2010-10-05 20:59:59,384][DEBUG][action.index] [Abominatrix] [index][0], node[7cfee996-5fc0-4048-8137-b812ac0c7c5f], [P], s[STARTED]: Failed to execute [index {[index][type] [1b080d59-7b60-4b98-97ba-e1b9b492a5e9], source[? execute]}] org.elasticsearch.ElasticSearchParseException: Failed to derive xcontent from [-128, 1, 0, 1, 0, 0, 0, 7, 101, 120, 101, 99, 117, 116, 101, 0, 0, 0, 0, 12, 0, 1, 8, 0, 1, 0, 0, 0, 2, 11, 0, 2, 0, 0, 0, 11, 47, 105, 110, 100, 101, 120, 47, 116, 121, 112, 101, 13, 0, 3, 11, 11, 0, 0, 0, 1, 0, 0, 0, 6, 112, 114, 101, 116, 116, 121, 0, 0, 0, 4, 116, 114, 117, 101, 13, 0, 4, 11, 11, 0, 0, 0, 1, 0, 0, 0, 12, 67, 111, 110, 116, 101, 110, 116, 45, 84, 121, 112, 101, 0, 0, 0, 16, 97, 112, 112, 108, 105, 99, 97, 116, 105, 111, 110, 47, 120, 115, 111, 110, 11, 0, 5, 0, 0, 0, 17, 123, 34, 110, 97, 109, 101, 34, 58, 32, 34, 118, 97, 108, 117, 101, 34, 125, 0, 0] at org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory.java: 132) at org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory.java: 123) at org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XContentDocumentMapper.java: 321) at org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XContentDocumentMapper.java: 308) at org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java: 208) at org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java: 140) at org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java: 62) at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction $AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java: 385) at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction $AsyncShardOperationAction.access $400(TransportShardReplicationOperationAction.java:212) at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction $AsyncShardOperationAction $1.run(TransportShardReplicationOperationAction.java:282) at java.util.concurrent.ThreadPoolExecutor $Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) [2010-10-05 21:01:10,418][DEBUG][action.index] [Abominatrix] [index][3], node[7cfee996-5fc0-4048-8137-b812ac0c7c5f], [P], s[STARTED]: Failed to execute [index {[index][type] [2627f2c1-9c79-4727-8154-99895d8933d2], source[? execute]}] org.elasticsearch.ElasticSearchParseException: Failed to derive xcontent from [-128, 1, 0, 1, 0, 0, 0, 7, 101, 120, 101, 99, 117, 116, 101, 0, 0, 0, 0, 12, 0, 1, 8, 0, 1, 0, 0, 0, 2, 11, 0, 2, 0, 0, 0, 11, 47, 105, 110, 100, 101, 120, 47, 116, 121, 112, 101, 13, 0, 3, 11, 11, 0, 0, 0, 1, 0, 0, 0, 6, 112, 114, 101, 116, 116, 121, 0, 0, 0, 4, 116, 114, 117, 101, 13, 0, 4, 11, 11, 0, 0, 0, 1, 0, 0, 0, 12, 67, 111, 110, 116, 101, 110, 116, 45, 84, 121, 112, 101, 0, 0, 0, 16, 97, 112, 112, 108, 105, 99, 97, 116, 105, 111, 110, 47, 106, 115, 111, 110, 11, 0, 5, 0, 0, 0, 17, 123, 34, 110, 97, 109, 101, 34, 58, 32, 34, 118, 97, 108, 117, 101, 34, 125, 0, 0] at org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory.java: 132) at org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory.java: 123) at org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XContentDocumentMapper.java: 321) at org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XContentDocumentMapper.java: 308) at org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java: 208) at org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java: 140) at org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java: 62) at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction $AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java: 385) at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction $AsyncShardOperationAction.access $400(TransportShardReplicationOperationAction.java:212) at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction $AsyncShardOperationAction $1.run(TransportShardReplicationOperationAction.java:282) at java.util.concurrent.ThreadPoolExecutor $Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) [2010-10-05 21:01:33,458][DEBUG][action.index] [Abominatrix] [index][3], node[7cfee996-5fc0-4048-8137-b812ac0c7c5f], [P], s[STARTED]: Failed to execute [index {[index][type][9317b3d8- ecdf-464c-840a-c6cfbc7e0c87], source[? execute]}] org.elasticsearch.ElasticSearchParseException: Failed to derive xcontent from [-128, 1, 0, 1, 0, 0, 0, 7, 101, 120, 101, 99, 117, 116, 101, 0, 0, 0, 0, 12, 0, 1, 8, 0, 1, 0, 0, 0, 2, 11, 0, 2, 0, 0, 0, 11, 47, 105, 110, 100, 101, 120, 47, 116, 121, 112, 101, 13, 0, 3, 11, 11, 0, 0, 0, 1, 0, 0, 0, 6, 112, 114, 101, 116, 116, 121, 0, 0, 0, 4, 116, 114, 117, 101, 13, 0, 4, 11, 11, 0, 0, 0, 1, 0, 0, 0, 12, 67, 111, 110, 116, 101, 110, 116, 45, 84, 121, 112, 101, 0, 0, 0, 16, 97, 112, 112, 108, 105, 99, 97, 116, 105, 111, 110, 47, 106, 115, 111, 110, 11, 0, 5, 0, 0, 0, 17, 123, 34, 110, 97, 109, 101, 34, 58, 32, 34, 118, 97, 108, 117, 101, 34, 125, 0, 0] at org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory.java: 132) at org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory.java: 123) at org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XContentDocumentMapper.java: 321) at org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XContentDocumentMapper.java: 308) at org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java: 208) at org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java: 140) at org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java: 62) at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction $AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java: 385) at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction $AsyncShardOperationAction.access $400(TransportShardReplicationOperationAction.java:212) at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction $AsyncShardOperationAction $1.run(TransportShardReplicationOperationAction.java:282) at java.util.concurrent.ThreadPoolExecutor $Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) On 10月5日, 下午8时48分, Medcl [me...@163.com](mailto:me...@163.com) wrote: > yes，and here is the code： > TSocket tsocket=new TSocket("localhost",9500); > var transport = new TFramedTransport(tsocket); > var protocol = new TBinaryProtocol(transport); > var client = new Rest.Client(protocol); > transport.Open(); > var restRequest = new RestRequest(); > restRequest.Method = Method.POST; > restRequest.Uri = "/index/type"; > restRequest.Body=System.Text.UTF8Encoding.UTF8.GetBytes("{"name": > "value"}"); > > ``` > restRequest.Parameters=new Dictionary(); > restRequest.Parameters.Add("pretty", "true"); > var response= client.execute(restRequest); > var str= System.Text.UTF8Encoding.Default.GetString(response.Body); > > ``` > > On 10月5日, 下午8时30分, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > Are you indexing a json? do you convert the json string into UTF8 byte > > array? > > > 2010/10/5 Medcl [me...@163.com](mailto:me...@163.com) > > > > hey,shay.banon ，i am using thrift in .net,and the type of request's > > > body and response's body is byte array，Method with Get is fine，and > > > the post method returned the following error，and i am a little > > > confused aboutxcontentand smile， i think i should use the right type > > > to serialize request.body or deserialize the response.body,so what > > > can i do？: > > > > [2010-10-05 20:05:39,999][DEBUG][action.index] > > > [Abominatrix] [index > > > ][2], node[7cfee996-5fc0-4048-8137-b812ac0c7c5f], [P], s[STARTED]: > > > Failed to exe > > > cute [index {[index][type][bc40f94b-aba4-43c8-a7d3-26517140ccd0], > > > source[? > > > > 1, 34, 125, 0, 0] > > > at > > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFa > > > ctory.java:132) > > > at > > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFa > > > ctory.java:123) > > > at > > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse( > > > XContentDocumentMapper.java:321) > > > at > > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse( > > > XContentDocumentMapper.java:308) > > > at > > > org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreat > > > e(InternalIndexShard.java:208) > > > at > > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnP > > > rimary(TransportIndexAction.java:140) > > > at > > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnP > > > rimary(TransportIndexAction.java:62) > > > at > > > org.elasticsearch.action.support.replication.TransportShardReplicatio > > > nOperationAction > > > $AsyncShardOperationAction.performOnPrimary(TransportShardReplic > > > ationOperationAction.java:385) > > > at > > > org.elasticsearch.action.support.replication.TransportShardReplicatio > > > nOperationAction$AsyncShardOperationAction.access > > > $400(TransportShardReplicationO > > > perationAction.java:212) > > > at > > > org.elasticsearch.action.support.replication.TransportShardReplicatio > > > nOperationAction$AsyncShardOperationAction > > > $1.run(TransportShardReplicationOperat > > > ionAction.java:282) > > > at java.util.concurrent.ThreadPoolExecutor > > > $Worker.runTask(ThreadPoolExec > > > utor.java:886) > > > at java.util.concurrent.ThreadPoolExecutor > > > $Worker.run(ThreadPoolExecutor > > > .java:908) > > > at java.lang.Thread.run(Thread.java:619) > > > On 9月15日, 下午7时00分, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > > > > > Hi, > > > > > ``` > > > > elasticsearch has support a binary format similar to json for some > > > > > > > > ``` > > > > > > time > > > > > > > now, named "xson". This was used internally when using the Java API when > > > > possible (for example, a search request used it automatically). The good > > > > news is that the good people at Jackson (actually Tatu) has created an > > > > excellent binary format for JSON called Smile. I have already pushed > > > > support > > > > for that, moreover, I have removed XSON completely and now use Smile as > > > > the > > > > formal binary format for JSON. > > > > > Another good news is that the Smile format is public and properly > > > > documented. You can find it here: > > > > [http://wiki.fasterxml.com/JacksonBinaryFormat](http://wiki.fasterxml.com/JacksonBinaryFormat), with the format here: > > > > [http://wiki.fasterxml.com/JacksonBinaryFormatSpec](http://wiki.fasterxml.com/JacksonBinaryFormatSpec). I must say that this is > > > > one of the best binary formats I have seen for JSON (BSON is an > > > > abomination). It is a bit complicated in some places (zigzagging numbers, > > > > and so on), but well worth the effort. > > > > > I am posting it here since if someone from other lang camps here can > > > > maybe take it and implement it in the respective language. I am trying to > > > > work with Tatu to get a proper web site to it and to make it feel more > > > > formal. This _should_ be the formal binary representation of JSON. > > > > > As far as elasticsearch, it plays with both easily, on all layers > > > > (including REST). Moreover, you can push data in one format and get it in > > > > another format easily (for example, JSON for debugging, but Smile as the > > > > default). With the REST layer, you don't have to provide content type, as > > > > its detected automatically based on the body, or you can simply sent a > > > > content type of application/smile for API level, source level format is > > > > automatically detected. Also, the response is automatically send using > > > > the > > > > format of the request. > > > > > On the Java end, there has been some refacoring which should not > > > > affect > > > > most users. If you were using the XContentBuilder base class, then > > > > nothing > > > > changes. The BinaryXContentBuilder and StringXContentBuilder were removed > > > > and now we only have XContentBuilder. The XSON format has basically been > > > > renamed to Smile format. > > > > > As a side note, elasticsearch has an abstraction on top of its both > > > > parsing and building of "xcontent", so, for example, support other > > > > fomrats > > > > (like XML) should not be too difficult to add. Are you interested in it? > > > > > -shay.banon ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [October 5, 2010, 1:49pm UTC](https://discuss.elastic.co/t/json-binary-format/3338/6 "2010-10-05T13:49:00Z")
Hey, Thats really strange, seems like the binary payload when trying to index is not what you sent. There can be several places where this might get messed up, both in elasticsearch and "outside" of it. Can we start with the basics? Can you print/gist the bytes generated (restRequest.Body) before you send the request? -shay.banon 2010/10/5 Medcl [medcl@163.com](mailto:medcl@163.com) > logs： > [2010-10-05 20:59:10,094][DEBUG][action.index] > [Abominatrix] [index][1], node[7cfee996-5fc0-4048-8137-b812ac0c7c5f], > [P], s[STARTED]: Failed to execute [index {[index][type][821ce2ad- > e20f-4d59-ba77-f75c820dd94e], source[? execute]}] > org.elasticsearch.ElasticSearchParseException: Failed to derive > xcontent from [-128, 1, 0, 1, 0, 0, 0, 7, 101, 120, 101, 99, 117, 116, > 101, 0, 0, 0, 0, 12, 0, 1, 8, 0, 1, 0, 0, 0, 2, 11, 0, 2, 0, 0, 0, 11, > 47, 105, 110, 100, 101, 120, 47, 116, 121, 112, 101, 13, 0, 3, 11, 11, > 0, 0, 0, 1, 0, 0, 0, 6, 112, 114, 101, 116, 116, 121, 0, 0, 0, 4, 116, > 114, 117, 101, 13, 0, 4, 11, 11, 0, 0, 0, 1, 0, 0, 0, 12, 67, 111, > 110, 116, 101, 110, 116, 45, 84, 121, 112, 101, 0, 0, 0, 16, 97, 112, > 112, 108, 105, 99, 97, 116, 105, 111, 110, 47, 120, 115, 111, 110, 11, > 0, 5, 0, 0, 0, 17, 123, 34, 110, 97, 109, 101, 34, 58, 32, 34, 118, > 97, 108, 117, 101, 34, 125, 0, 0] > at > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory.java: > 132) > at > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory.java: > 123) > at > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XContentDocumentMapper.java: > 321) > at > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XContentDocumentMapper.java: > 308) > at > > org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java: > 208) > at > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java: > 140) > at > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java: > 62) > at > > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction > > $AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java: > 385) > at > > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction > $AsyncShardOperationAction.access > $400(TransportShardReplicationOperationAction.java:212) > at > > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction > $AsyncShardOperationAction > $1.run(TransportShardReplicationOperationAction.java:282) > at java.util.concurrent.ThreadPoolExecutor > $Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor > $Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > [2010-10-05 20:59:59,384][DEBUG][action.index] > [Abominatrix] [index][0], node[7cfee996-5fc0-4048-8137-b812ac0c7c5f], > [P], s[STARTED]: Failed to execute [index {[index][type] > [1b080d59-7b60-4b98-97ba-e1b9b492a5e9], source[? execute]}] > org.elasticsearch.ElasticSearchParseException: Failed to derive > xcontent from [-128, 1, 0, 1, 0, 0, 0, 7, 101, 120, 101, 99, 117, 116, > 101, 0, 0, 0, 0, 12, 0, 1, 8, 0, 1, 0, 0, 0, 2, 11, 0, 2, 0, 0, 0, 11, > 47, 105, 110, 100, 101, 120, 47, 116, 121, 112, 101, 13, 0, 3, 11, 11, > 0, 0, 0, 1, 0, 0, 0, 6, 112, 114, 101, 116, 116, 121, 0, 0, 0, 4, 116, > 114, 117, 101, 13, 0, 4, 11, 11, 0, 0, 0, 1, 0, 0, 0, 12, 67, 111, > 110, 116, 101, 110, 116, 45, 84, 121, 112, 101, 0, 0, 0, 16, 97, 112, > 112, 108, 105, 99, 97, 116, 105, 111, 110, 47, 120, 115, 111, 110, 11, > 0, 5, 0, 0, 0, 17, 123, 34, 110, 97, 109, 101, 34, 58, 32, 34, 118, > 97, 108, 117, 101, 34, 125, 0, 0] > at > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory.java: > 132) > at > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory.java: > 123) > at > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XContentDocumentMapper.java: > 321) > at > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XContentDocumentMapper.java: > 308) > at > > org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java: > 208) > at > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java: > 140) > at > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java: > 62) > at > > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction > > $AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java: > 385) > at > > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction > $AsyncShardOperationAction.access > $400(TransportShardReplicationOperationAction.java:212) > at > > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction > $AsyncShardOperationAction > $1.run(TransportShardReplicationOperationAction.java:282) > at java.util.concurrent.ThreadPoolExecutor > $Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor > $Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > [2010-10-05 21:01:10,418][DEBUG][action.index] > [Abominatrix] [index][3], node[7cfee996-5fc0-4048-8137-b812ac0c7c5f], > [P], s[STARTED]: Failed to execute [index {[index][type] > [2627f2c1-9c79-4727-8154-99895d8933d2], source[? execute]}] > org.elasticsearch.ElasticSearchParseException: Failed to derive > xcontent from [-128, 1, 0, 1, 0, 0, 0, 7, 101, 120, 101, 99, 117, 116, > 101, 0, 0, 0, 0, 12, 0, 1, 8, 0, 1, 0, 0, 0, 2, 11, 0, 2, 0, 0, 0, 11, > 47, 105, 110, 100, 101, 120, 47, 116, 121, 112, 101, 13, 0, 3, 11, 11, > 0, 0, 0, 1, 0, 0, 0, 6, 112, 114, 101, 116, 116, 121, 0, 0, 0, 4, 116, > 114, 117, 101, 13, 0, 4, 11, 11, 0, 0, 0, 1, 0, 0, 0, 12, 67, 111, > 110, 116, 101, 110, 116, 45, 84, 121, 112, 101, 0, 0, 0, 16, 97, 112, > 112, 108, 105, 99, 97, 116, 105, 111, 110, 47, 106, 115, 111, 110, 11, > 0, 5, 0, 0, 0, 17, 123, 34, 110, 97, 109, 101, 34, 58, 32, 34, 118, > 97, 108, 117, 101, 34, 125, 0, 0] > at > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory.java: > 132) > at > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory.java: > 123) > at > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XContentDocumentMapper.java: > 321) > at > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XContentDocumentMapper.java: > 308) > at > > org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java: > 208) > at > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java: > 140) > at > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java: > 62) > at > > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction > > $AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java: > 385) > at > > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction > $AsyncShardOperationAction.access > $400(TransportShardReplicationOperationAction.java:212) > at > > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction > $AsyncShardOperationAction > $1.run(TransportShardReplicationOperationAction.java:282) > at java.util.concurrent.ThreadPoolExecutor > $Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor > $Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > [2010-10-05 21:01:33,458][DEBUG][action.index] > [Abominatrix] [index][3], node[7cfee996-5fc0-4048-8137-b812ac0c7c5f], > [P], s[STARTED]: Failed to execute [index {[index][type][9317b3d8- > ecdf-464c-840a-c6cfbc7e0c87], source[? execute]}] > org.elasticsearch.ElasticSearchParseException: Failed to derive > xcontent from [-128, 1, 0, 1, 0, 0, 0, 7, 101, 120, 101, 99, 117, 116, > 101, 0, 0, 0, 0, 12, 0, 1, 8, 0, 1, 0, 0, 0, 2, 11, 0, 2, 0, 0, 0, 11, > 47, 105, 110, 100, 101, 120, 47, 116, 121, 112, 101, 13, 0, 3, 11, 11, > 0, 0, 0, 1, 0, 0, 0, 6, 112, 114, 101, 116, 116, 121, 0, 0, 0, 4, 116, > 114, 117, 101, 13, 0, 4, 11, 11, 0, 0, 0, 1, 0, 0, 0, 12, 67, 111, > 110, 116, 101, 110, 116, 45, 84, 121, 112, 101, 0, 0, 0, 16, 97, 112, > 112, 108, 105, 99, 97, 116, 105, 111, 110, 47, 106, 115, 111, 110, 11, > 0, 5, 0, 0, 0, 17, 123, 34, 110, 97, 109, 101, 34, 58, 32, 34, 118, > 97, 108, 117, 101, 34, 125, 0, 0] > at > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory.java: > 132) > at > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory.java: > 123) > at > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XContentDocumentMapper.java: > 321) > at > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XContentDocumentMapper.java: > 308) > at > > org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java: > 208) > at > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java: > 140) > at > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java: > 62) > at > > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction > > $AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java: > 385) > at > > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction > $AsyncShardOperationAction.access > $400(TransportShardReplicationOperationAction.java:212) > at > > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction > $AsyncShardOperationAction > $1.run(TransportShardReplicationOperationAction.java:282) > at java.util.concurrent.ThreadPoolExecutor > $Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor > $Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > > On 10月5日, 下午8时48分, Medcl [me...@163.com](mailto:me...@163.com) wrote: > > > yes，and here is the code： > > TSocket tsocket=new TSocket("localhost",9500); > > var transport = new TFramedTransport(tsocket); > > var protocol = new TBinaryProtocol(transport); > > var client = new Rest.Client(protocol); > > transport.Open(); > > var restRequest = new RestRequest(); > > restRequest.Method = Method.POST; > > restRequest.Uri = "/index/type"; > > restRequest.Body=System.Text.UTF8Encoding.UTF8.GetBytes("{"name": > > > "value"}"); > > > > ``` > > restRequest.Parameters=new Dictionary > > > ``` > > string\>(); > > > ``` > > restRequest.Parameters.Add("pretty", "true"); > > var response= client.execute(restRequest); > > var str= > > > > ``` > > System.Text.UTF8Encoding.Default.GetString(response.Body); > > > On 10月5日, 下午8时30分, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > > > Are you indexing a json? do you convert the json string into UTF8 byte > > > array? > > > > > 2010/10/5 Medcl [me...@163.com](mailto:me...@163.com) > > > > > > hey,shay.banon ，i am using thrift in .net,and the type of request's > > > > body and response's body is byte array，Method with Get is fine，and > > > > the post method returned the following error，and i am a little > > > > confused aboutxcontentand smile， i think i should use the right type > > > > to serialize request.body or deserialize the response.body,so what > > > > can i do？: > > > > > > [2010-10-05 20:05:39,999][DEBUG][action.index] > > > > [Abominatrix] [index > > > > ][2], node[7cfee996-5fc0-4048-8137-b812ac0c7c5f], [P], s[STARTED]: > > > > Failed to exe > > > > cute [index {[index][type][bc40f94b-aba4-43c8-a7d3-26517140ccd0], > > > > source[? > > > > > > 1, 34, 125, 0, 0] > > > > at > > > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFa > > > > ctory.java:132) > > > > at > > > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFa > > > > ctory.java:123) > > > > at > > > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse( > > > > XContentDocumentMapper.java:321) > > > > at > > > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse( > > > > XContentDocumentMapper.java:308) > > > > at > > > > org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreat > > > > e(InternalIndexShard.java:208) > > > > at > > > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnP > > > > rimary(TransportIndexAction.java:140) > > > > at > > > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnP > > > > rimary(TransportIndexAction.java:62) > > > > at > > > > org.elasticsearch.action.support.replication.TransportShardReplicatio > > > > nOperationAction > > > > $AsyncShardOperationAction.performOnPrimary(TransportShardReplic > > > > ationOperationAction.java:385) > > > > at > > > > org.elasticsearch.action.support.replication.TransportShardReplicatio > > > > nOperationAction$AsyncShardOperationAction.access > > > > $400(TransportShardReplicationO > > > > perationAction.java:212) > > > > at > > > > org.elasticsearch.action.support.replication.TransportShardReplicatio > > > > nOperationAction$AsyncShardOperationAction > > > > $1.run(TransportShardReplicationOperat > > > > ionAction.java:282) > > > > at java.util.concurrent.ThreadPoolExecutor > > > > $Worker.runTask(ThreadPoolExec > > > > utor.java:886) > > > > at java.util.concurrent.ThreadPoolExecutor > > > > $Worker.run(ThreadPoolExecutor > > > > .java:908) > > > > at java.lang.Thread.run(Thread.java:619) > > > > On 9月15日, 下午7时00分, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > > > > > > > Hi, > > > > > > > ``` > > > > > elasticsearch has support a binary format similar to json for > > > > > > > > > > ``` > > some > > > > > time > > > > > > > > > now, named "xson". This was used internally when using the Java API > > > > > when > > > > > possible (for example, a search request used it automatically). The > > > > > good > > > > > news is that the good people at Jackson (actually Tatu) has created > > > > > an > > > > > excellent binary format for JSON called Smile. I have already > > > > > pushed > > > > > support > > > > > for that, moreover, I have removed XSON completely and now use > > > > > Smile as > > > > > the > > > > > formal binary format for JSON. > > > > > > > Another good news is that the Smile format is public and > > > > > properly > > > > > documented. You can find it here: > > > > > [http://wiki.fasterxml.com/JacksonBinaryFormat](http://wiki.fasterxml.com/JacksonBinaryFormat), with the format here: > > > > > [http://wiki.fasterxml.com/JacksonBinaryFormatSpec](http://wiki.fasterxml.com/JacksonBinaryFormatSpec). I must say that > > > > > this is > > > > > one of the best binary formats I have seen for JSON (BSON is an > > > > > abomination). It is a bit complicated in some places (zigzagging > > > > > numbers, > > > > > and so on), but well worth the effort. > > > > > > > I am posting it here since if someone from other lang camps here > > > > > can > > > > > maybe take it and implement it in the respective language. I am > > > > > trying to > > > > > work with Tatu to get a proper web site to it and to make it feel > > > > > more > > > > > formal. This _should_ be the formal binary representation of JSON. > > > > > > > As far as elasticsearch, it plays with both easily, on all > > > > > layers > > > > > (including REST). Moreover, you can push data in one format and get > > > > > it in > > > > > another format easily (for example, JSON for debugging, but Smile > > > > > as the > > > > > default). With the REST layer, you don't have to provide content > > > > > type, as > > > > > its detected automatically based on the body, or you can simply > > > > > sent a > > > > > content type of application/smile for API level, source level > > > > > format is > > > > > automatically detected. Also, the response is automatically send > > > > > using > > > > > the > > > > > format of the request. > > > > > > > On the Java end, there has been some refacoring which should not > > > > > affect > > > > > most users. If you were using the XContentBuilder base class, then > > > > > nothing > > > > > changes. The BinaryXContentBuilder and StringXContentBuilder were > > > > > removed > > > > > and now we only have XContentBuilder. The XSON format has basically > > > > > been > > > > > renamed to Smile format. > > > > > > > As a side note, elasticsearch has an abstraction on top of its > > > > > both > > > > > parsing and building of "xcontent", so, for example, support other > > > > > fomrats > > > > > (like XML) should not be too difficult to add. Are you interested > > > > > in it? > > > > > > > -shay.banon ---
**Author:** ![medcl\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/medcl_2/32/3297_2.png) [@medcl\_2](https://discuss.elastic.co/u/medcl_2)\ **Post date:** [October 6, 2010, 2:07am UTC](https://discuss.elastic.co/t/json-binary-format/3338/7 "2010-10-06T02:07:09Z")
the response's status is INTERNAL\_SERVER\_ERROR and here is the body's bytearray restRequest.Body = System.Text.UTF8Encoding.Default.GetBytes("{"a":"b" }"); - ``` restRequest.Body {byte[10]} byte[] [0] 123 byte [1] 34 byte [2] 97 byte [3] 34 byte [4] 58 byte [5] 34 byte [6] 98 byte [7] 34 byte [8] 32 byte [9] 125 byte ``` these bytes can be back to my string（{"a":"b"}） by use System.Text.UTF8Encoding.Default.GetString(restRequest.Body); var str = System.Text.UTF8Encoding.Default.GetString(response.Body); and the response's body is: str: { "error" : "ReplicationShardOperationFailedException[[index][3] ]; nested: ElasticSearchParseException[Failed to derive xcontent from [-128, 1, 0, 1, 0, 0, 0, 7, 101, 120, 101, 99, 117, 116, 101, 0, 0, 0, 0, 12, 0, 1, 8, 0, 1, 0, 0, 0, 2, 11, 0, 2, 0, 0, 0, 11, 47, 105, 110, 100, 101, 120, 47, 116, 121, 112, 101, 13, 0, 3, 11, 11, 0, 0, 0, 1, 0, 0, 0, 6, 112, 114, 101, 116, 116, 121, 0, 0, 0, 4, 116, 114, 117, 101, 13, 0, 4, 11, 11, 0, 0, 0, 1, 0, 0, 0, 12, 67, 111, 110, 116, 101, 110, 116, 45, 84, 121, 112, 101, 0, 0, 0, 16, 97, 112, 112, 108, 105, 99, 97, 116, 105, 111, 110, 47, 106, 115, 111, 110, 11, 0, 5, 0, 0, 0, 10, 123, 34, 97, 34, 58, 34, 98, 34, 32, 125, 0, 0]]; " } On 10月5日, 下午9时49分, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > Hey, > > Thats really strange, seems like the binary payload when trying to index > is not what you sent. There can be several places where this might get > messed up, both in elasticsearch and "outside" of it. Can we start with the > basics? Can you print/gist the bytes generated (restRequest.Body) before you > send the request? > > -shay.banon > > 2010/10/5 Medcl [me...@163.com](mailto:me...@163.com) > > > logs： > > [2010-10-05 20:59:10,094][DEBUG][action.index] > > [Abominatrix] [index][1], node[7cfee996-5fc0-4048-8137-b812ac0c7c5f], > > [P], s[STARTED]: Failed to execute [index {[index][type][821ce2ad- > > e20f-4d59-ba77-f75c820dd94e], source[? execute]}] > > org.elasticsearch.ElasticSearchParseException: Failed to derive > > xcontent from [-128, 1, 0, 1, 0, 0, 0, 7, 101, 120, 101, 99, 117, 116, > > 101, 0, 0, 0, 0, 12, 0, 1, 8, 0, 1, 0, 0, 0, 2, 11, 0, 2, 0, 0, 0, 11, > > 47, 105, 110, 100, 101, 120, 47, 116, 121, 112, 101, 13, 0, 3, 11, 11, > > 0, 0, 0, 1, 0, 0, 0, 6, 112, 114, 101, 116, 116, 121, 0, 0, 0, 4, 116, > > 114, 117, 101, 13, 0, 4, 11, 11, 0, 0, 0, 1, 0, 0, 0, 12, 67, 111, > > 110, 116, 101, 110, 116, 45, 84, 121, 112, 101, 0, 0, 0, 16, 97, 112, > > 112, 108, 105, 99, 97, 116, 105, 111, 110, 47, 120, 115, 111, 110, 11, > > 0, 5, 0, 0, 0, 17, 123, 34, 110, 97, 109, 101, 34, 58, 32, 34, 118, > > 97, 108, 117, 101, 34, 125, 0, 0] > > at > > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory. java: > > 132) > > at > > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory. java: > > 123) > > at > > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XConte ntDocumentMapper.java: > > 321) > > at > > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XConte ntDocumentMapper.java: > > 308) > > at > > > org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(Inte rnalIndexShard.java: > > 208) > > at > > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary (TransportIndexAction.java: > > 140) > > at > > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary (TransportIndexAction.java: > > 62) > > at > > > org.elasticsearch.action.support.replication.TransportShardReplicationOpera tionAction > > > $AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperat ionAction.java: > > 385) > > at > > > org.elasticsearch.action.support.replication.TransportShardReplicationOpera tionAction > > $AsyncShardOperationAction.access > > $400(TransportShardReplicationOperationAction.java:212) > > at > > > org.elasticsearch.action.support.replication.TransportShardReplicationOpera tionAction > > $AsyncShardOperationAction > > $1.run(TransportShardReplicationOperationAction.java:282) > > at java.util.concurrent.ThreadPoolExecutor > > $Worker.runTask(ThreadPoolExecutor.java:886) > > at java.util.concurrent.ThreadPoolExecutor > > $Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:619) > > [2010-10-05 20:59:59,384][DEBUG][action.index] > > [Abominatrix] [index][0], node[7cfee996-5fc0-4048-8137-b812ac0c7c5f], > > [P], s[STARTED]: Failed to execute [index {[index][type] > > [1b080d59-7b60-4b98-97ba-e1b9b492a5e9], source[? execute]}] > > org.elasticsearch.ElasticSearchParseException: Failed to derive > > xcontent from [-128, 1, 0, 1, 0, 0, 0, 7, 101, 120, 101, 99, 117, 116, > > 101, 0, 0, 0, 0, 12, 0, 1, 8, 0, 1, 0, 0, 0, 2, 11, 0, 2, 0, 0, 0, 11, > > 47, 105, 110, 100, 101, 120, 47, 116, 121, 112, 101, 13, 0, 3, 11, 11, > > 0, 0, 0, 1, 0, 0, 0, 6, 112, 114, 101, 116, 116, 121, 0, 0, 0, 4, 116, > > 114, 117, 101, 13, 0, 4, 11, 11, 0, 0, 0, 1, 0, 0, 0, 12, 67, 111, > > 110, 116, 101, 110, 116, 45, 84, 121, 112, 101, 0, 0, 0, 16, 97, 112, > > 112, 108, 105, 99, 97, 116, 105, 111, 110, 47, 120, 115, 111, 110, 11, > > 0, 5, 0, 0, 0, 17, 123, 34, 110, 97, 109, 101, 34, 58, 32, 34, 118, > > 97, 108, 117, 101, 34, 125, 0, 0] > > at > > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory. java: > > 132) > > at > > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory. java: > > 123) > > at > > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XConte ntDocumentMapper.java: > > 321) > > at > > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XConte ntDocumentMapper.java: > > 308) > > at > > > org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(Inte rnalIndexShard.java: > > 208) > > at > > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary (TransportIndexAction.java: > > 140) > > at > > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary (TransportIndexAction.java: > > 62) > > at > > > org.elasticsearch.action.support.replication.TransportShardReplicationOpera tionAction > > > $AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperat ionAction.java: > > 385) > > at > > > org.elasticsearch.action.support.replication.TransportShardReplicationOpera tionAction > > $AsyncShardOperationAction.access > > $400(TransportShardReplicationOperationAction.java:212) > > at > > > org.elasticsearch.action.support.replication.TransportShardReplicationOpera tionAction > > $AsyncShardOperationAction > > $1.run(TransportShardReplicationOperationAction.java:282) > > at java.util.concurrent.ThreadPoolExecutor > > $Worker.runTask(ThreadPoolExecutor.java:886) > > at java.util.concurrent.ThreadPoolExecutor > > $Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:619) > > [2010-10-05 21:01:10,418][DEBUG][action.index] > > [Abominatrix] [index][3], node[7cfee996-5fc0-4048-8137-b812ac0c7c5f], > > [P], s[STARTED]: Failed to execute [index {[index][type] > > [2627f2c1-9c79-4727-8154-99895d8933d2], source[? execute]}] > > org.elasticsearch.ElasticSearchParseException: Failed to derive > > xcontent from [-128, 1, 0, 1, 0, 0, 0, 7, 101, 120, 101, 99, 117, 116, > > 101, 0, 0, 0, 0, 12, 0, 1, 8, 0, 1, 0, 0, 0, 2, 11, 0, 2, 0, 0, 0, 11, > > 47, 105, 110, 100, 101, 120, 47, 116, 121, 112, 101, 13, 0, 3, 11, 11, > > 0, 0, 0, 1, 0, 0, 0, 6, 112, 114, 101, 116, 116, 121, 0, 0, 0, 4, 116, > > 114, 117, 101, 13, 0, 4, 11, 11, 0, 0, 0, 1, 0, 0, 0, 12, 67, 111, > > 110, 116, 101, 110, 116, 45, 84, 121, 112, 101, 0, 0, 0, 16, 97, 112, > > 112, 108, 105, 99, 97, 116, 105, 111, 110, 47, 106, 115, 111, 110, 11, > > 0, 5, 0, 0, 0, 17, 123, 34, 110, 97, 109, 101, 34, 58, 32, 34, 118, > > 97, 108, 117, 101, 34, 125, 0, 0] > > at > > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory. java: > > 132) > > at > > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory. java: > > 123) > > at > > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XConte ntDocumentMapper.java: > > 321) > > at > > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XConte ntDocumentMapper.java: > > 308) > > at > > > org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(Inte rnalIndexShard.java: > > 208) > > at > > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary (TransportIndexAction.java: > > 140) > > at > > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary (TransportIndexAction.java: > > 62) > > at > > > org.elasticsearch.action.support.replication.TransportShardReplicationOpera tionAction > > > $AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperat ionAction.java: > > 385) > > at > > > org.elasticsearch.action.support.replication.TransportShardReplicationOpera tionAction > > $AsyncShardOperationAction.access > > $400(TransportShardReplicationOperationAction.java:212) > > at > > > org.elasticsearch.action.support.replication.TransportShardReplicationOpera tionAction > > $AsyncShardOperationAction > > $1.run(TransportShardReplicationOperationAction.java:282) > > at java.util.concurrent.ThreadPoolExecutor > > $Worker.runTask(ThreadPoolExecutor.java:886) > > at java.util.concurrent.ThreadPoolExecutor > > $Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:619) > > [2010-10-05 21:01:33,458][DEBUG][action.index] > > [Abominatrix] [index][3], node[7cfee996-5fc0-4048-8137-b812ac0c7c5f], > > [P], s[STARTED]: Failed to execute [index {[index][type][9317b3d8- > > ecdf-464c-840a-c6cfbc7e0c87], source[? execute]}] > > org.elasticsearch.ElasticSearchParseException: Failed to derive > > xcontent from [-128, 1, 0, 1, 0, 0, 0, 7, 101, 120, 101, 99, 117, 116, > > 101, 0, 0, 0, 0, 12, 0, 1, 8, 0, 1, 0, 0, 0, 2, 11, 0, 2, 0, 0, 0, 11, > > 47, 105, 110, 100, 101, 120, 47, 116, 121, 112, 101, 13, 0, 3, 11, 11, > > 0, 0, 0, 1, 0, 0, 0, 6, 112, 114, 101, 116, 116, 121, 0, 0, 0, 4, 116, > > 114, 117, 101, 13, 0, 4, 11, 11, 0, 0, 0, 1, 0, 0, 0, 12, 67, 111, > > 110, 116, 101, 110, 116, 45, 84, 121, 112, 101, 0, 0, 0, 16, 97, 112, > > 112, 108, 105, 99, 97, 116, 105, 111, 110, 47, 106, 115, 111, 110, 11, > > 0, 5, 0, 0, 0, 17, 123, 34, 110, 97, 109, 101, 34, 58, 32, 34, 118, > > 97, 108, 117, 101, 34, 125, 0, 0] > > at > > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory. java: > > 132) > > at > > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory. java: > > 123) > > at > > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XConte ntDocumentMapper.java: > > 321) > > at > > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XConte ntDocumentMapper.java: > > 308) > > at > > > org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(Inte rnalIndexShard.java: > > 208) > > at > > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary (TransportIndexAction.java: > > 140) > > at > > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary (TransportIndexAction.java: > > 62) > > at > > ... > > 阅读更多 \>\> ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [October 6, 2010, 2:39pm UTC](https://discuss.elastic.co/t/json-binary-format/3338/8 "2010-10-06T14:39:53Z")
ok, it looks like within the bytes on elasticsearch side, there is some data before the actual content. The big question is where its coming from (the exception is thrown during the indexing of the documents). Is there a chance that you can get on IRC in a few hours, and I will build some debug version of elasticsearch to see where its coming from? Which version of thrift do you use? -shay.banon 2010/10/6 Medcl [medcl@163.com](mailto:medcl@163.com) > the response's status is INTERNAL\_SERVER\_ERROR > and here is the body's bytearray > restRequest.Body = > System.Text.UTF8Encoding.Default.GetBytes("{"a":"b" }"); > > - > > ``` > restRequest.Body > {byte[10]} byte[] > [0] 123 byte > [1] 34 byte > [2] 97 byte > [3] 34 byte > [4] 58 byte > [5] 34 byte > [6] 98 byte > [7] 34 byte > [8] 32 byte > [9] 125 byte > > ``` > > these bytes can be back to my string（{"a":"b"}） by use > System.Text.UTF8Encoding.Default.GetString(restRequest.Body); > > var str = System.Text.UTF8Encoding.Default.GetString(response.Body); > and the response's body is: > str: > { > "error" : "ReplicationShardOperationFailedException[[index][3] ]; > nested: ElasticSearchParseException[Failed to derive xcontent from > [-128, 1, 0, 1, 0, 0, 0, 7, 101, 120, 101, 99, 117, 116, 101, 0, 0, 0, > 0, 12, 0, 1, 8, 0, 1, 0, 0, 0, 2, 11, 0, 2, 0, 0, 0, 11, 47, 105, 110, > 100, 101, 120, 47, 116, 121, 112, 101, 13, 0, 3, 11, 11, 0, 0, 0, 1, > 0, 0, 0, 6, 112, 114, 101, 116, 116, 121, 0, 0, 0, 4, 116, 114, 117, > 101, 13, 0, 4, 11, 11, 0, 0, 0, 1, 0, 0, 0, 12, 67, 111, 110, 116, > 101, 110, 116, 45, 84, 121, 112, 101, 0, 0, 0, 16, 97, 112, 112, 108, > 105, 99, 97, 116, 105, 111, 110, 47, 106, 115, 111, 110, 11, 0, 5, 0, > 0, 0, 10, 123, 34, 97, 34, 58, 34, 98, 34, 32, 125, 0, 0]]; " > } > > On 10月5日, 下午9时49分, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > Hey, > > > > Thats really strange, seems like the binary payload when trying to > > index > > is not what you sent. There can be several places where this might get > > messed up, both in elasticsearch and "outside" of it. Can we start with > > the > > basics? Can you print/gist the bytes generated (restRequest.Body) before > > you > > send the request? > > > > -shay.banon > > > > 2010/10/5 Medcl [me...@163.com](mailto:me...@163.com) > > > > > logs： > > > [2010-10-05 20:59:10,094][DEBUG][action.index] > > > [Abominatrix] [index][1], node[7cfee996-5fc0-4048-8137-b812ac0c7c5f], > > > [P], s[STARTED]: Failed to execute [index {[index][type][821ce2ad- > > > e20f-4d59-ba77-f75c820dd94e], source[? execute]}] > > > org.elasticsearch.ElasticSearchParseException: Failed to derive > > > xcontent from [-128, 1, 0, 1, 0, 0, 0, 7, 101, 120, 101, 99, 117, 116, > > > 101, 0, 0, 0, 0, 12, 0, 1, 8, 0, 1, 0, 0, 0, 2, 11, 0, 2, 0, 0, 0, 11, > > > 47, 105, 110, 100, 101, 120, 47, 116, 121, 112, 101, 13, 0, 3, 11, 11, > > > 0, 0, 0, 1, 0, 0, 0, 6, 112, 114, 101, 116, 116, 121, 0, 0, 0, 4, 116, > > > 114, 117, 101, 13, 0, 4, 11, 11, 0, 0, 0, 1, 0, 0, 0, 12, 67, 111, > > > 110, 116, 101, 110, 116, 45, 84, 121, 112, 101, 0, 0, 0, 16, 97, 112, > > > 112, 108, 105, 99, 97, 116, 105, 111, 110, 47, 120, 115, 111, 110, 11, > > > 0, 5, 0, 0, 0, 17, 123, 34, 110, 97, 109, 101, 34, 58, 32, 34, 118, > > > 97, 108, 117, 101, 34, 125, 0, 0] > > > at > > > > > > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory. > java: > > > > 1. at > > > > > > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory. > java: > > > > 1. at > > > > > > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XConte > ntDocumentMapper.java: > > > > 1. at > > > > > > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XConte > ntDocumentMapper.java: > > > > 1. at > > > > > > > org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(Inte > rnalIndexShard.java: > > > > 1. at > > > > > > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary > (TransportIndexAction.java: > > > > 1. at > > > > > > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary > (TransportIndexAction.java: > > > > 1. at > > > > > > > org.elasticsearch.action.support.replication.TransportShardReplicationOpera > tionAction > > > > > > $AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperat > ionAction.java: > > > > 1. at > > > > > > > org.elasticsearch.action.support.replication.TransportShardReplicationOpera > tionAction > > > > $AsyncShardOperationAction.access > > > $400(TransportShardReplicationOperationAction.java:212) > > > at > > > > > > > org.elasticsearch.action.support.replication.TransportShardReplicationOpera > tionAction > > > > $AsyncShardOperationAction > > > $1.run(TransportShardReplicationOperationAction.java:282) > > > at java.util.concurrent.ThreadPoolExecutor > > > $Worker.runTask(ThreadPoolExecutor.java:886) > > > at java.util.concurrent.ThreadPoolExecutor > > > $Worker.run(ThreadPoolExecutor.java:908) > > > at java.lang.Thread.run(Thread.java:619) > > > [2010-10-05 20:59:59,384][DEBUG][action.index] > > > [Abominatrix] [index][0], node[7cfee996-5fc0-4048-8137-b812ac0c7c5f], > > > [P], s[STARTED]: Failed to execute [index {[index][type] > > > [1b080d59-7b60-4b98-97ba-e1b9b492a5e9], source[? execute]}] > > > org.elasticsearch.ElasticSearchParseException: Failed to derive > > > xcontent from [-128, 1, 0, 1, 0, 0, 0, 7, 101, 120, 101, 99, 117, 116, > > > 101, 0, 0, 0, 0, 12, 0, 1, 8, 0, 1, 0, 0, 0, 2, 11, 0, 2, 0, 0, 0, 11, > > > 47, 105, 110, 100, 101, 120, 47, 116, 121, 112, 101, 13, 0, 3, 11, 11, > > > 0, 0, 0, 1, 0, 0, 0, 6, 112, 114, 101, 116, 116, 121, 0, 0, 0, 4, 116, > > > 114, 117, 101, 13, 0, 4, 11, 11, 0, 0, 0, 1, 0, 0, 0, 12, 67, 111, > > > 110, 116, 101, 110, 116, 45, 84, 121, 112, 101, 0, 0, 0, 16, 97, 112, > > > 112, 108, 105, 99, 97, 116, 105, 111, 110, 47, 120, 115, 111, 110, 11, > > > 0, 5, 0, 0, 0, 17, 123, 34, 110, 97, 109, 101, 34, 58, 32, 34, 118, > > > 97, 108, 117, 101, 34, 125, 0, 0] > > > at > > > > > > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory. > java: > > > > 1. at > > > > > > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory. > java: > > > > 1. at > > > > > > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XConte > ntDocumentMapper.java: > > > > 1. at > > > > > > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XConte > ntDocumentMapper.java: > > > > 1. at > > > > > > > org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(Inte > rnalIndexShard.java: > > > > 1. at > > > > > > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary > (TransportIndexAction.java: > > > > 1. at > > > > > > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary > (TransportIndexAction.java: > > > > 1. at > > > > > > > org.elasticsearch.action.support.replication.TransportShardReplicationOpera > tionAction > > > > > > $AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperat > ionAction.java: > > > > 1. at > > > > > > > org.elasticsearch.action.support.replication.TransportShardReplicationOpera > tionAction > > > > $AsyncShardOperationAction.access > > > $400(TransportShardReplicationOperationAction.java:212) > > > at > > > > > > > org.elasticsearch.action.support.replication.TransportShardReplicationOpera > tionAction > > > > $AsyncShardOperationAction > > > $1.run(TransportShardReplicationOperationAction.java:282) > > > at java.util.concurrent.ThreadPoolExecutor > > > $Worker.runTask(ThreadPoolExecutor.java:886) > > > at java.util.concurrent.ThreadPoolExecutor > > > $Worker.run(ThreadPoolExecutor.java:908) > > > at java.lang.Thread.run(Thread.java:619) > > > [2010-10-05 21:01:10,418][DEBUG][action.index] > > > [Abominatrix] [index][3], node[7cfee996-5fc0-4048-8137-b812ac0c7c5f], > > > [P], s[STARTED]: Failed to execute [index {[index][type] > > > [2627f2c1-9c79-4727-8154-99895d8933d2], source[? execute]}] > > > org.elasticsearch.ElasticSearchParseException: Failed to derive > > > xcontent from [-128, 1, 0, 1, 0, 0, 0, 7, 101, 120, 101, 99, 117, 116, > > > 101, 0, 0, 0, 0, 12, 0, 1, 8, 0, 1, 0, 0, 0, 2, 11, 0, 2, 0, 0, 0, 11, > > > 47, 105, 110, 100, 101, 120, 47, 116, 121, 112, 101, 13, 0, 3, 11, 11, > > > 0, 0, 0, 1, 0, 0, 0, 6, 112, 114, 101, 116, 116, 121, 0, 0, 0, 4, 116, > > > 114, 117, 101, 13, 0, 4, 11, 11, 0, 0, 0, 1, 0, 0, 0, 12, 67, 111, > > > 110, 116, 101, 110, 116, 45, 84, 121, 112, 101, 0, 0, 0, 16, 97, 112, > > > 112, 108, 105, 99, 97, 116, 105, 111, 110, 47, 106, 115, 111, 110, 11, > > > 0, 5, 0, 0, 0, 17, 123, 34, 110, 97, 109, 101, 34, 58, 32, 34, 118, > > > 97, 108, 117, 101, 34, 125, 0, 0] > > > at > > > > > > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory. > java: > > > > 1. at > > > > > > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory. > java: > > > > 1. at > > > > > > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XConte > ntDocumentMapper.java: > > > > 1. at > > > > > > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XConte > ntDocumentMapper.java: > > > > 1. at > > > > > > > org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(Inte > rnalIndexShard.java: > > > > 1. at > > > > > > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary > (TransportIndexAction.java: > > > > 1. at > > > > > > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary > (TransportIndexAction.java: > > > > 1. at > > > > > > > org.elasticsearch.action.support.replication.TransportShardReplicationOpera > tionAction > > > > > > $AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperat > ionAction.java: > > > > 1. at > > > > > > > org.elasticsearch.action.support.replication.TransportShardReplicationOpera > tionAction > > > > $AsyncShardOperationAction.access > > > $400(TransportShardReplicationOperationAction.java:212) > > > at > > > > > > > org.elasticsearch.action.support.replication.TransportShardReplicationOpera > tionAction > > > > $AsyncShardOperationAction > > > $1.run(TransportShardReplicationOperationAction.java:282) > > > at java.util.concurrent.ThreadPoolExecutor > > > $Worker.runTask(ThreadPoolExecutor.java:886) > > > at java.util.concurrent.ThreadPoolExecutor > > > $Worker.run(ThreadPoolExecutor.java:908) > > > at java.lang.Thread.run(Thread.java:619) > > > [2010-10-05 21:01:33,458][DEBUG][action.index] > > > [Abominatrix] [index][3], node[7cfee996-5fc0-4048-8137-b812ac0c7c5f], > > > [P], s[STARTED]: Failed to execute [index {[index][type][9317b3d8- > > > ecdf-464c-840a-c6cfbc7e0c87], source[? execute]}] > > > org.elasticsearch.ElasticSearchParseException: Failed to derive > > > xcontent from [-128, 1, 0, 1, 0, 0, 0, 7, 101, 120, 101, 99, 117, 116, > > > 101, 0, 0, 0, 0, 12, 0, 1, 8, 0, 1, 0, 0, 0, 2, 11, 0, 2, 0, 0, 0, 11, > > > 47, 105, 110, 100, 101, 120, 47, 116, 121, 112, 101, 13, 0, 3, 11, 11, > > > 0, 0, 0, 1, 0, 0, 0, 6, 112, 114, 101, 116, 116, 121, 0, 0, 0, 4, 116, > > > 114, 117, 101, 13, 0, 4, 11, 11, 0, 0, 0, 1, 0, 0, 0, 12, 67, 111, > > > 110, 116, 101, 110, 116, 45, 84, 121, 112, 101, 0, 0, 0, 16, 97, 112, > > > 112, 108, 105, 99, 97, 116, 105, 111, 110, 47, 106, 115, 111, 110, 11, > > > 0, 5, 0, 0, 0, 17, 123, 34, 110, 97, 109, 101, 34, 58, 32, 34, 118, > > > 97, 108, 117, 101, 34, 125, 0, 0] > > > at > > > > > > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory. > java: > > > > 1. at > > > > > > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory. > java: > > > > 1. at > > > > > > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XConte > ntDocumentMapper.java: > > > > 1. at > > > > > > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XConte > ntDocumentMapper.java: > > > > 1. at > > > > > > > org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(Inte > rnalIndexShard.java: > > > > 1. at > > > > > > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary > (TransportIndexAction.java: > > > > 1. at > > > > > > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary > (TransportIndexAction.java: > > > > 1. at > > > > ... > > > > 阅读更多 \>\> ---
**Author:** ![medcl\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/medcl_2/32/3297_2.png) [@medcl\_2](https://discuss.elastic.co/u/medcl_2)\ **Post date:** [October 7, 2010, 9:15am UTC](https://discuss.elastic.co/t/json-binary-format/3338/9 "2010-10-07T09:15:26Z")
sorry for my later reply,and my thrift version is 0.4.0,i use the manual build version(0.12.0 snapshot) ,does this effects the data that send to elasticsearch? when will you be the IRC channel? i will on the channel when i am online 🙂 On 10月6日, 下午10时39分, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > ok, it looks like within the bytes on elasticsearch side, there is some data > before the actual content. The big question is where its coming from (the > exception is thrown during the indexing of the documents). Is there a chance > that you can get on IRC in a few hours, and I will build some debug version > of elasticsearch to see where its coming from? > > Which version of thrift do you use? > > -shay.banon > > 2010/10/6 Medcl [me...@163.com](mailto:me...@163.com) > > > the response's status is INTERNAL\_SERVER\_ERROR > > and here is the body's bytearray > > restRequest.Body = > > System.Text.UTF8Encoding.Default.GetBytes("{"a":"b" }"); > > > - > > > > ``` > > restRequest.Body > > {byte[10]} byte[] > > [0] 123 byte > > [1] 34 byte > > [2] 97 byte > > [3] 34 byte > > [4] 58 byte > > [5] 34 byte > > [6] 98 byte > > [7] 34 byte > > [8] 32 byte > > [9] 125 byte > > > > ``` > > > > these bytes can be back to my string（{"a":"b"}） by use > > System.Text.UTF8Encoding.Default.GetString(restRequest.Body); > > > var str = System.Text.UTF8Encoding.Default.GetString(response.Body); > > and the response's body is: > > str: > > { > > "error" : "ReplicationShardOperationFailedException[[index][3] ]; > > nested: ElasticSearchParseException[Failed to derive xcontent from > > [-128, 1, 0, 1, 0, 0, 0, 7, 101, 120, 101, 99, 117, 116, 101, 0, 0, 0, > > 0, 12, 0, 1, 8, 0, 1, 0, 0, 0, 2, 11, 0, 2, 0, 0, 0, 11, 47, 105, 110, > > 100, 101, 120, 47, 116, 121, 112, 101, 13, 0, 3, 11, 11, 0, 0, 0, 1, > > 0, 0, 0, 6, 112, 114, 101, 116, 116, 121, 0, 0, 0, 4, 116, 114, 117, > > 101, 13, 0, 4, 11, 11, 0, 0, 0, 1, 0, 0, 0, 12, 67, 111, 110, 116, > > 101, 110, 116, 45, 84, 121, 112, 101, 0, 0, 0, 16, 97, 112, 112, 108, > > 105, 99, 97, 116, 105, 111, 110, 47, 106, 115, 111, 110, 11, 0, 5, 0, > > 0, 0, 10, 123, 34, 97, 34, 58, 34, 98, 34, 32, 125, 0, 0]]; " > > } > > > On 10月5日, 下午9时49分, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > > > Hey, > > > > Thats really strange, seems like the binary payload when trying to > > > index > > > is not what you sent. There can be several places where this might get > > > messed up, both in elasticsearch and "outside" of it. Can we start with > > > the > > > basics? Can you print/gist the bytes generated (restRequest.Body) before > > > you > > > send the request? > > > > -shay.banon > > > > 2010/10/5 Medcl [me...@163.com](mailto:me...@163.com) > > > > > logs： > > > > [2010-10-05 20:59:10,094][DEBUG][action.index] > > > > [Abominatrix] [index][1], node[7cfee996-5fc0-4048-8137-b812ac0c7c5f], > > > > [P], s[STARTED]: Failed to execute [index {[index][type][821ce2ad- > > > > e20f-4d59-ba77-f75c820dd94e], source[? execute]}] > > > > org.elasticsearch.ElasticSearchParseException: Failed to derive > > > > xcontent from [-128, 1, 0, 1, 0, 0, 0, 7, 101, 120, 101, 99, 117, 116, > > > > 101, 0, 0, 0, 0, 12, 0, 1, 8, 0, 1, 0, 0, 0, 2, 11, 0, 2, 0, 0, 0, 11, > > > > 47, 105, 110, 100, 101, 120, 47, 116, 121, 112, 101, 13, 0, 3, 11, 11, > > > > 0, 0, 0, 1, 0, 0, 0, 6, 112, 114, 101, 116, 116, 121, 0, 0, 0, 4, 116, > > > > 114, 117, 101, 13, 0, 4, 11, 11, 0, 0, 0, 1, 0, 0, 0, 12, 67, 111, > > > > 110, 116, 101, 110, 116, 45, 84, 121, 112, 101, 0, 0, 0, 16, 97, 112, > > > > 112, 108, 105, 99, 97, 116, 105, 111, 110, 47, 120, 115, 111, 110, 11, > > > > 0, 5, 0, 0, 0, 17, 123, 34, 110, 97, 109, 101, 34, 58, 32, 34, 118, > > > > 97, 108, 117, 101, 34, 125, 0, 0] > > > > at > > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory. > > java: > > > > > > 1. at > > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory. > > java: > > > > > > 1. at > > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XConte > > ntDocumentMapper.java: > > > > > > 1. at > > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XConte > > ntDocumentMapper.java: > > > > > > 1. at > > > org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(Inte > > rnalIndexShard.java: > > > > > > 1. at > > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary > > (TransportIndexAction.java: > > > > > > 1. at > > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary > > (TransportIndexAction.java: > > > > > > 1. at > > > org.elasticsearch.action.support.replication.TransportShardReplicationOpera > > tionAction > > > $AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperat > > ionAction.java: > > > > > > 1. at > > > org.elasticsearch.action.support.replication.TransportShardReplicationOpera > > tionAction > > > > > > $AsyncShardOperationAction.access > > > > $400(TransportShardReplicationOperationAction.java:212) > > > > at > > > org.elasticsearch.action.support.replication.TransportShardReplicationOpera > > tionAction > > > > > > $AsyncShardOperationAction > > > > $1.run(TransportShardReplicationOperationAction.java:282) > > > > at java.util.concurrent.ThreadPoolExecutor > > > > $Worker.runTask(ThreadPoolExecutor.java:886) > > > > at java.util.concurrent.ThreadPoolExecutor > > > > $Worker.run(ThreadPoolExecutor.java:908) > > > > at java.lang.Thread.run(Thread.java:619) > > > > [2010-10-05 20:59:59,384][DEBUG][action.index] > > > > [Abominatrix] [index][0], node[7cfee996-5fc0-4048-8137-b812ac0c7c5f], > > > > [P], s[STARTED]: Failed to execute [index {[index][type] > > > > [1b080d59-7b60-4b98-97ba-e1b9b492a5e9], source[? execute]}] > > > > org.elasticsearch.ElasticSearchParseException: Failed to derive > > > > xcontent from [-128, 1, 0, 1, 0, 0, 0, 7, 101, 120, 101, 99, 117, 116, > > > > 101, 0, 0, 0, 0, 12, 0, 1, 8, 0, 1, 0, 0, 0, 2, 11, 0, 2, 0, 0, 0, 11, > > > > 47, 105, 110, 100, 101, 120, 47, 116, 121, 112, 101, 13, 0, 3, 11, 11, > > > > 0, 0, 0, 1, 0, 0, 0, 6, 112, 114, 101, 116, 116, 121, 0, 0, 0, 4, 116, > > > > 114, 117, 101, 13, 0, 4, 11, 11, 0, 0, 0, 1, 0, 0, 0, 12, 67, 111, > > > > 110, 116, 101, 110, 116, 45, 84, 121, 112, 101, 0, 0, 0, 16, 97, 112, > > > > 112, 108, 105, 99, 97, 116, 105, 111, 110, 47, 120, 115, 111, 110, 11, > > > > 0, 5, 0, 0, 0, 17, 123, 34, 110, 97, 109, 101, 34, 58, 32, 34, 118, > > > > 97, 108, 117, 101, 34, 125, 0, 0] > > > > at > > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory. > > java: > > > > > > 1. at > > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory. > > java: > > > > > > 1. at > > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XConte > > ntDocumentMapper.java: > > > > > > 1. at > > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XConte > > ntDocumentMapper.java: > > > > > > 1. at > > > org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(Inte > > rnalIndexShard.java: > > > > > > 1. at > > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary > > (TransportIndexAction.java: > > > > > > 1. at > > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary > > (TransportIndexAction.java: > > > > > > 1. at > > > org.elasticsearch.action.support.replication.TransportShardReplicationOpera > > tionAction > > > $AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperat > > ionAction.java: > > > > > > 1. at > > > org.elasticsearch.action.support.replication.TransportShardReplicationOpera > > tionAction > > > > > > $AsyncShardOperationAction.access > > > > $400(TransportShardReplicationOperationAction.java:212) > > > > at > > > org.elasticsearch.action.support.replication.TransportShardReplicationOpera > > tionAction > > > > > > $AsyncShardOperationAction > > > > $1.run(TransportShardReplicationOperationAction.java:282) > > > > at java.util.concurrent.ThreadPoolExecutor > > > > $Worker.runTask(ThreadPoolExecutor.java:886) > > > > at java.util.concurrent.ThreadPoolExecutor > > > > $Worker.run(ThreadPoolExecutor.java:908) > > > > at java.lang.Thread.run(Thread.java:619) > > > > [2010-10-05 21:01:10,418][DEBUG][action.index] > > > > [Abominatrix] [index][3], node[7cfee996-5fc0-4048-8137-b812ac0c7c5f], > > > > [P], s[STARTED]: Failed to execute [index {[index][type] > > > > [2627f2c1-9c79-4727-8154-99895d8933d2], source[? execute]}] > > > > org.elasticsearch.ElasticSearchParseException: Failed to derive > > > > xcontent from [-128, 1, 0, 1, 0, 0, 0, 7, 101, 120, 101, 99, 117, 116, > > > > 101, 0, 0, 0, 0, 12, 0, 1, 8, 0, 1, 0, 0, 0, 2, 11, 0, 2, 0, 0, 0, 11, > > > > 47, 105, 110, 100, 101, 120, 47, 116, 121, 112, 101, 13, 0, 3, 11, 11, > > > > 0, 0, 0, 1, 0, 0, 0, 6, 112, 114, 101, 116, 116, 121, 0, 0, 0, 4, 116, > > > > 114, 117, 101, 13, 0, 4, 11, 11, 0, 0, 0, 1, 0, 0, 0, 12, 67, 111, > > > > 110, 116, 101, 110, 116, 45, 84, 121, 112, 101, 0, 0, 0, 16, 97, 112, > > > > 112, 108, 105, 99, 97, 116, 105, 111, 110, 47, 106, 115, 111, 110, 11, > > > > 0, 5, 0, 0, 0, 17, 123, 34, 110, 97, 109, 101, 34, 58, 32, 34, 118, > > > > 97, 108, 117, 101, 34, 125, 0, 0] > > > > at > > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory. > > java: > > > > > > 1. at > > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory. > > java: > > > > > > 1. at > > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XConte > > ntDocumentMapper.java: > > > > > > 1. at > > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XConte > > ntDocumentMapper.java: > > > > > > 1. at > > > org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(Inte > > rnalIndexShard.java: > > > > > > 1. at > > ... > > 阅读更多 \>\> ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [October 7, 2010, 9:39am UTC](https://discuss.elastic.co/t/json-binary-format/3338/10 "2010-10-07T09:39:07Z")
Great, wanted to check that both server and client are on 0.4 (things have changed in thrift). Will catch up on IRC. 2010/10/7 Medcl [medcl@163.com](mailto:medcl@163.com) > sorry for my later reply,and my thrift version is 0.4.0,i use the > manual build version(0.12.0 snapshot) ,does this effects the data that > send to elasticsearch? when will you be the IRC channel? i will on the > channel when i am online 🙂 > > On 10月6日, 下午10时39分, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > ok, it looks like within the bytes on elasticsearch side, there is some > > data > > before the actual content. The big question is where its coming from (the > > exception is thrown during the indexing of the documents). Is there a > > chance > > that you can get on IRC in a few hours, and I will build some debug > > version > > of elasticsearch to see where its coming from? > > > > Which version of thrift do you use? > > > > -shay.banon > > > > 2010/10/6 Medcl [me...@163.com](mailto:me...@163.com) > > > > > the response's status is INTERNAL\_SERVER\_ERROR > > > and here is the body's bytearray > > > restRequest.Body = > > > System.Text.UTF8Encoding.Default.GetBytes("{"a":"b" }"); > > > > > - > > > > > > ``` > > > restRequest.Body > > > {byte[10]} byte[] > > > [0] 123 byte > > > [1] 34 byte > > > [2] 97 byte > > > [3] 34 byte > > > [4] 58 byte > > > [5] 34 byte > > > [6] 98 byte > > > [7] 34 byte > > > [8] 32 byte > > > [9] 125 byte > > > > > > ``` > > > > > > these bytes can be back to my string（{"a":"b"}） by use > > > System.Text.UTF8Encoding.Default.GetString(restRequest.Body); > > > > > var str = System.Text.UTF8Encoding.Default.GetString(response.Body); > > > and the response's body is: > > > str: > > > { > > > "error" : "ReplicationShardOperationFailedException[[index][3] ]; > > > nested: ElasticSearchParseException[Failed to derive xcontent from > > > [-128, 1, 0, 1, 0, 0, 0, 7, 101, 120, 101, 99, 117, 116, 101, 0, 0, 0, > > > 0, 12, 0, 1, 8, 0, 1, 0, 0, 0, 2, 11, 0, 2, 0, 0, 0, 11, 47, 105, 110, > > > 100, 101, 120, 47, 116, 121, 112, 101, 13, 0, 3, 11, 11, 0, 0, 0, 1, > > > 0, 0, 0, 6, 112, 114, 101, 116, 116, 121, 0, 0, 0, 4, 116, 114, 117, > > > 101, 13, 0, 4, 11, 11, 0, 0, 0, 1, 0, 0, 0, 12, 67, 111, 110, 116, > > > 101, 110, 116, 45, 84, 121, 112, 101, 0, 0, 0, 16, 97, 112, 112, 108, > > > 105, 99, 97, 116, 105, 111, 110, 47, 106, 115, 111, 110, 11, 0, 5, 0, > > > 0, 0, 10, 123, 34, 97, 34, 58, 34, 98, 34, 32, 125, 0, 0]]; " > > > } > > > > > On 10月5日, 下午9时49分, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > > > > > Hey, > > > > > > Thats really strange, seems like the binary payload when trying to > > > > index > > > > is not what you sent. There can be several places where this might > > > > get > > > > messed up, both in elasticsearch and "outside" of it. Can we start > > > > with > > > > the > > > > basics? Can you print/gist the bytes generated (restRequest.Body) > > > > before > > > > you > > > > send the request? > > > > > > -shay.banon > > > > > > 2010/10/5 Medcl [me...@163.com](mailto:me...@163.com) > > > > > > > logs： > > > > > [2010-10-05 20:59:10,094][DEBUG][action.index] > > > > > [Abominatrix] [index][1], > > > > > node[7cfee996-5fc0-4048-8137-b812ac0c7c5f], > > > > > [P], s[STARTED]: Failed to execute [index {[index][type][821ce2ad- > > > > > e20f-4d59-ba77-f75c820dd94e], source[? execute]}] > > > > > org.elasticsearch.ElasticSearchParseException: Failed to derive > > > > > xcontent from [-128, 1, 0, 1, 0, 0, 0, 7, 101, 120, 101, 99, 117, > > > > > 116, > > > > > 101, 0, 0, 0, 0, 12, 0, 1, 8, 0, 1, 0, 0, 0, 2, 11, 0, 2, 0, 0, 0, > > > > > 11, > > > > > 47, 105, 110, 100, 101, 120, 47, 116, 121, 112, 101, 13, 0, 3, 11, > > > > > 11, > > > > > 0, 0, 0, 1, 0, 0, 0, 6, 112, 114, 101, 116, 116, 121, 0, 0, 0, 4, > > > > > 116, > > > > > 114, 117, 101, 13, 0, 4, 11, 11, 0, 0, 0, 1, 0, 0, 0, 12, 67, 111, > > > > > 110, 116, 101, 110, 116, 45, 84, 121, 112, 101, 0, 0, 0, 16, 97, > > > > > 112, > > > > > 112, 108, 105, 99, 97, 116, 105, 111, 110, 47, 120, 115, 111, 110, > > > > > 11, > > > > > 0, 5, 0, 0, 0, 17, 123, 34, 110, 97, 109, 101, 34, 58, 32, 34, 118, > > > > > 97, 108, 117, 101, 34, 125, 0, 0] > > > > > at > > > > > > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory. > > > > java: > > > > > > > > 1. at > > > > > > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory. > > > > java: > > > > > > > > 1. at > > > > > > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XConte > > > > ntDocumentMapper.java: > > > > > > > > 1. at > > > > > > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XConte > > > > ntDocumentMapper.java: > > > > > > > > 1. at > > > > > > > org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(Inte > > > > rnalIndexShard.java: > > > > > > > > 1. at > > > > > > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary > > > > (TransportIndexAction.java: > > > > > > > > 1. at > > > > > > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary > > > > (TransportIndexAction.java: > > > > > > > > 1. at > > > > > > > org.elasticsearch.action.support.replication.TransportShardReplicationOpera > > > > tionAction > > > > > > > $AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperat > > > > ionAction.java: > > > > > > > > 1. at > > > > > > > org.elasticsearch.action.support.replication.TransportShardReplicationOpera > > > > tionAction > > > > > > > > $AsyncShardOperationAction.access > > > > > $400(TransportShardReplicationOperationAction.java:212) > > > > > at > > > > > > > org.elasticsearch.action.support.replication.TransportShardReplicationOpera > > > > tionAction > > > > > > > > $AsyncShardOperationAction > > > > > $1.run(TransportShardReplicationOperationAction.java:282) > > > > > at java.util.concurrent.ThreadPoolExecutor > > > > > $Worker.runTask(ThreadPoolExecutor.java:886) > > > > > at java.util.concurrent.ThreadPoolExecutor > > > > > $Worker.run(ThreadPoolExecutor.java:908) > > > > > at java.lang.Thread.run(Thread.java:619) > > > > > [2010-10-05 20:59:59,384][DEBUG][action.index] > > > > > [Abominatrix] [index][0], > > > > > node[7cfee996-5fc0-4048-8137-b812ac0c7c5f], > > > > > [P], s[STARTED]: Failed to execute [index {[index][type] > > > > > [1b080d59-7b60-4b98-97ba-e1b9b492a5e9], source[? execute]}] > > > > > org.elasticsearch.ElasticSearchParseException: Failed to derive > > > > > xcontent from [-128, 1, 0, 1, 0, 0, 0, 7, 101, 120, 101, 99, 117, > > > > > 116, > > > > > 101, 0, 0, 0, 0, 12, 0, 1, 8, 0, 1, 0, 0, 0, 2, 11, 0, 2, 0, 0, 0, > > > > > 11, > > > > > 47, 105, 110, 100, 101, 120, 47, 116, 121, 112, 101, 13, 0, 3, 11, > > > > > 11, > > > > > 0, 0, 0, 1, 0, 0, 0, 6, 112, 114, 101, 116, 116, 121, 0, 0, 0, 4, > > > > > 116, > > > > > 114, 117, 101, 13, 0, 4, 11, 11, 0, 0, 0, 1, 0, 0, 0, 12, 67, 111, > > > > > 110, 116, 101, 110, 116, 45, 84, 121, 112, 101, 0, 0, 0, 16, 97, > > > > > 112, > > > > > 112, 108, 105, 99, 97, 116, 105, 111, 110, 47, 120, 115, 111, 110, > > > > > 11, > > > > > 0, 5, 0, 0, 0, 17, 123, 34, 110, 97, 109, 101, 34, 58, 32, 34, 118, > > > > > 97, 108, 117, 101, 34, 125, 0, 0] > > > > > at > > > > > > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory. > > > > java: > > > > > > > > 1. at > > > > > > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory. > > > > java: > > > > > > > > 1. at > > > > > > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XConte > > > > ntDocumentMapper.java: > > > > > > > > 1. at > > > > > > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XConte > > > > ntDocumentMapper.java: > > > > > > > > 1. at > > > > > > > org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(Inte > > > > rnalIndexShard.java: > > > > > > > > 1. at > > > > > > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary > > > > (TransportIndexAction.java: > > > > > > > > 1. at > > > > > > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary > > > > (TransportIndexAction.java: > > > > > > > > 1. at > > > > > > > org.elasticsearch.action.support.replication.TransportShardReplicationOpera > > > > tionAction > > > > > > > $AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperat > > > > ionAction.java: > > > > > > > > 1. at > > > > > > > org.elasticsearch.action.support.replication.TransportShardReplicationOpera > > > > tionAction > > > > > > > > $AsyncShardOperationAction.access > > > > > $400(TransportShardReplicationOperationAction.java:212) > > > > > at > > > > > > > org.elasticsearch.action.support.replication.TransportShardReplicationOpera > > > > tionAction > > > > > > > > $AsyncShardOperationAction > > > > > $1.run(TransportShardReplicationOperationAction.java:282) > > > > > at java.util.concurrent.ThreadPoolExecutor > > > > > $Worker.runTask(ThreadPoolExecutor.java:886) > > > > > at java.util.concurrent.ThreadPoolExecutor > > > > > $Worker.run(ThreadPoolExecutor.java:908) > > > > > at java.lang.Thread.run(Thread.java:619) > > > > > [2010-10-05 21:01:10,418][DEBUG][action.index] > > > > > [Abominatrix] [index][3], > > > > > node[7cfee996-5fc0-4048-8137-b812ac0c7c5f], > > > > > [P], s[STARTED]: Failed to execute [index {[index][type] > > > > > [2627f2c1-9c79-4727-8154-99895d8933d2], source[? execute]}] > > > > > org.elasticsearch.ElasticSearchParseException: Failed to derive > > > > > xcontent from [-128, 1, 0, 1, 0, 0, 0, 7, 101, 120, 101, 99, 117, > > > > > 116, > > > > > 101, 0, 0, 0, 0, 12, 0, 1, 8, 0, 1, 0, 0, 0, 2, 11, 0, 2, 0, 0, 0, > > > > > 11, > > > > > 47, 105, 110, 100, 101, 120, 47, 116, 121, 112, 101, 13, 0, 3, 11, > > > > > 11, > > > > > 0, 0, 0, 1, 0, 0, 0, 6, 112, 114, 101, 116, 116, 121, 0, 0, 0, 4, > > > > > 116, > > > > > 114, 117, 101, 13, 0, 4, 11, 11, 0, 0, 0, 1, 0, 0, 0, 12, 67, 111, > > > > > 110, 116, 101, 110, 116, 45, 84, 121, 112, 101, 0, 0, 0, 16, 97, > > > > > 112, > > > > > 112, 108, 105, 99, 97, 116, 105, 111, 110, 47, 106, 115, 111, 110, > > > > > 11, > > > > > 0, 5, 0, 0, 0, 17, 123, 34, 110, 97, 109, 101, 34, 58, 32, 34, 118, > > > > > 97, 108, 117, 101, 34, 125, 0, 0] > > > > > at > > > > > > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory. > > > > java: > > > > > > > > 1. at > > > > > > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory. > > > > java: > > > > > > > > 1. at > > > > > > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XConte > > > > ntDocumentMapper.java: > > > > > > > > 1. at > > > > > > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XConte > > > > ntDocumentMapper.java: > > > > > > > > 1. at > > > > > > > org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(Inte > > > > rnalIndexShard.java: > > > > > > > > 1. at > > > > ... > > > > 阅读更多 \>\> ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [October 7, 2010, 11:16am UTC](https://discuss.elastic.co/t/json-binary-format/3338/11 "2010-10-07T11:16:45Z")
Hi, Just an update (to the list) after the debug session on IRC. It seems like thrift framed transport is problematic. In elasticsearch, you can start it in four modes (thrift.type): threadpool (not framed), threadpool\_framed (framed), nonblocking (framed - has to be), hsha (framed, has to be). If the client and the server are on different protocol (framed vs. not framed), then obscure problem happens. Its importnat for the client and the server to use the same framed, or non framed, protocol. The first problem was that the .net client was using framed transport, and the server (es) was using the default threadpool (which is not framed). Once we moved to .net client to be non framed, it worked. Then, we tried moving es to be framed (threadpool\_framed), and the client framed, but it still did not work. Seems like a bug in thrift framing support. Not sure where, smells like the .NET side. If someone is using thrift in other lang, would love to hear if it works in framed mode for them (make sure its an API that uses the body, like index). Thanks, shay.banon 2010/10/7 Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) > Great, wanted to check that both server and client are on 0.4 (things have > changed in thrift). Will catch up on IRC. > > 2010/10/7 Medcl [medcl@163.com](mailto:medcl@163.com) > > sorry for my later reply,and my thrift version is 0.4.0,i use the > > > manual build version(0.12.0 snapshot) ,does this effects the data that > > send to elasticsearch? when will you be the IRC channel? i will on the > > channel when i am online 🙂 > > > > On 10月6日, 下午10时39分, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > > > ok, it looks like within the bytes on elasticsearch side, there is some > > > data > > > before the actual content. The big question is where its coming from > > > (the > > > exception is thrown during the indexing of the documents). Is there a > > > chance > > > that you can get on IRC in a few hours, and I will build some debug > > > version > > > of elasticsearch to see where its coming from? > > > > > > Which version of thrift do you use? > > > > > > -shay.banon > > > > > > 2010/10/6 Medcl [me...@163.com](mailto:me...@163.com) > > > > > > > the response's status is INTERNAL\_SERVER\_ERROR > > > > and here is the body's bytearray > > > > restRequest.Body = > > > > System.Text.UTF8Encoding.Default.GetBytes("{"a":"b" }"); > > > > > > > - > > > > > > > > ``` > > > > restRequest.Body > > > > {byte[10]} byte[] > > > > [0] 123 byte > > > > [1] 34 byte > > > > [2] 97 byte > > > > [3] 34 byte > > > > [4] 58 byte > > > > [5] 34 byte > > > > [6] 98 byte > > > > [7] 34 byte > > > > [8] 32 byte > > > > [9] 125 byte > > > > > > > > ``` > > > > > > > > these bytes can be back to my string（{"a":"b"}） by use > > > > System.Text.UTF8Encoding.Default.GetString(restRequest.Body); > > > > > > > var str = System.Text.UTF8Encoding.Default.GetString(response.Body); > > > > and the response's body is: > > > > str: > > > > { > > > > "error" : "ReplicationShardOperationFailedException[[index][3] ]; > > > > nested: ElasticSearchParseException[Failed to derive xcontent from > > > > [-128, 1, 0, 1, 0, 0, 0, 7, 101, 120, 101, 99, 117, 116, 101, 0, 0, 0, > > > > 0, 12, 0, 1, 8, 0, 1, 0, 0, 0, 2, 11, 0, 2, 0, 0, 0, 11, 47, 105, 110, > > > > 100, 101, 120, 47, 116, 121, 112, 101, 13, 0, 3, 11, 11, 0, 0, 0, 1, > > > > 0, 0, 0, 6, 112, 114, 101, 116, 116, 121, 0, 0, 0, 4, 116, 114, 117, > > > > 101, 13, 0, 4, 11, 11, 0, 0, 0, 1, 0, 0, 0, 12, 67, 111, 110, 116, > > > > 101, 110, 116, 45, 84, 121, 112, 101, 0, 0, 0, 16, 97, 112, 112, 108, > > > > 105, 99, 97, 116, 105, 111, 110, 47, 106, 115, 111, 110, 11, 0, 5, 0, > > > > 0, 0, 10, 123, 34, 97, 34, 58, 34, 98, 34, 32, 125, 0, 0]]; " > > > > } > > > > > > > On 10月5日, 下午9时49分, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > > > > > > > Hey, > > > > > > > > Thats really strange, seems like the binary payload when trying > > > > > to > > > > > index > > > > > is not what you sent. There can be several places where this might > > > > > get > > > > > messed up, both in elasticsearch and "outside" of it. Can we start > > > > > with > > > > > the > > > > > basics? Can you print/gist the bytes generated (restRequest.Body) > > > > > before > > > > > you > > > > > send the request? > > > > > > > > -shay.banon > > > > > > > > 2010/10/5 Medcl [me...@163.com](mailto:me...@163.com) > > > > > > > > > logs： > > > > > > [2010-10-05 20:59:10,094][DEBUG][action.index] > > > > > > [Abominatrix] [index][1], > > > > > > node[7cfee996-5fc0-4048-8137-b812ac0c7c5f], > > > > > > [P], s[STARTED]: Failed to execute [index {[index][type][821ce2ad- > > > > > > e20f-4d59-ba77-f75c820dd94e], source[? execute]}] > > > > > > org.elasticsearch.ElasticSearchParseException: Failed to derive > > > > > > xcontent from [-128, 1, 0, 1, 0, 0, 0, 7, 101, 120, 101, 99, 117, > > > > > > 116, > > > > > > 101, 0, 0, 0, 0, 12, 0, 1, 8, 0, 1, 0, 0, 0, 2, 11, 0, 2, 0, 0, 0, > > > > > > 11, > > > > > > 47, 105, 110, 100, 101, 120, 47, 116, 121, 112, 101, 13, 0, 3, 11, > > > > > > 11, > > > > > > 0, 0, 0, 1, 0, 0, 0, 6, 112, 114, 101, 116, 116, 121, 0, 0, 0, 4, > > > > > > 116, > > > > > > 114, 117, 101, 13, 0, 4, 11, 11, 0, 0, 0, 1, 0, 0, 0, 12, 67, 111, > > > > > > 110, 116, 101, 110, 116, 45, 84, 121, 112, 101, 0, 0, 0, 16, 97, > > > > > > 112, > > > > > > 112, 108, 105, 99, 97, 116, 105, 111, 110, 47, 120, 115, 111, 110, > > > > > > 11, > > > > > > 0, 5, 0, 0, 0, 17, 123, 34, 110, 97, 109, 101, 34, 58, 32, 34, > > > > > > 118, > > > > > > 97, 108, 117, 101, 34, 125, 0, 0] > > > > > > at > > > > > > > > > > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory. > > > > > > java: > > > > > > > > > > 1. at > > > > > > > > > > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory. > > > > > > java: > > > > > > > > > > 1. at > > > > > > > > > > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XConte > > > > > > ntDocumentMapper.java: > > > > > > > > > > 1. at > > > > > > > > > > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XConte > > > > > > ntDocumentMapper.java: > > > > > > > > > > 1. at > > > > > > > > > > > org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(Inte > > > > > > rnalIndexShard.java: > > > > > > > > > > 1. at > > > > > > > > > > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary > > > > > > (TransportIndexAction.java: > > > > > > > > > > 1. at > > > > > > > > > > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary > > > > > > (TransportIndexAction.java: > > > > > > > > > > 1. at > > > > > > > > > > > org.elasticsearch.action.support.replication.TransportShardReplicationOpera > > > > > > tionAction > > > > > > > > > > > $AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperat > > > > > > ionAction.java: > > > > > > > > > > 1. at > > > > > > > > > > > org.elasticsearch.action.support.replication.TransportShardReplicationOpera > > > > > > tionAction > > > > > > > > > > $AsyncShardOperationAction.access > > > > > > $400(TransportShardReplicationOperationAction.java:212) > > > > > > at > > > > > > > > > > > org.elasticsearch.action.support.replication.TransportShardReplicationOpera > > > > > > tionAction > > > > > > > > > > $AsyncShardOperationAction > > > > > > $1.run(TransportShardReplicationOperationAction.java:282) > > > > > > at java.util.concurrent.ThreadPoolExecutor > > > > > > $Worker.runTask(ThreadPoolExecutor.java:886) > > > > > > at java.util.concurrent.ThreadPoolExecutor > > > > > > $Worker.run(ThreadPoolExecutor.java:908) > > > > > > at java.lang.Thread.run(Thread.java:619) > > > > > > [2010-10-05 20:59:59,384][DEBUG][action.index] > > > > > > [Abominatrix] [index][0], > > > > > > node[7cfee996-5fc0-4048-8137-b812ac0c7c5f], > > > > > > [P], s[STARTED]: Failed to execute [index {[index][type] > > > > > > [1b080d59-7b60-4b98-97ba-e1b9b492a5e9], source[? execute > > > > > > ]}] > > > > > > org.elasticsearch.ElasticSearchParseException: Failed to derive > > > > > > xcontent from [-128, 1, 0, 1, 0, 0, 0, 7, 101, 120, 101, 99, 117, > > > > > > 116, > > > > > > 101, 0, 0, 0, 0, 12, 0, 1, 8, 0, 1, 0, 0, 0, 2, 11, 0, 2, 0, 0, 0, > > > > > > 11, > > > > > > 47, 105, 110, 100, 101, 120, 47, 116, 121, 112, 101, 13, 0, 3, 11, > > > > > > 11, > > > > > > 0, 0, 0, 1, 0, 0, 0, 6, 112, 114, 101, 116, 116, 121, 0, 0, 0, 4, > > > > > > 116, > > > > > > 114, 117, 101, 13, 0, 4, 11, 11, 0, 0, 0, 1, 0, 0, 0, 12, 67, 111, > > > > > > 110, 116, 101, 110, 116, 45, 84, 121, 112, 101, 0, 0, 0, 16, 97, > > > > > > 112, > > > > > > 112, 108, 105, 99, 97, 116, 105, 111, 110, 47, 120, 115, 111, 110, > > > > > > 11, > > > > > > 0, 5, 0, 0, 0, 17, 123, 34, 110, 97, 109, 101, 34, 58, 32, 34, > > > > > > 118, > > > > > > 97, 108, 117, 101, 34, 125, 0, 0] > > > > > > at > > > > > > > > > > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory. > > > > > > java: > > > > > > > > > > 1. at > > > > > > > > > > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory. > > > > > > java: > > > > > > > > > > 1. at > > > > > > > > > > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XConte > > > > > > ntDocumentMapper.java: > > > > > > > > > > 1. at > > > > > > > > > > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XConte > > > > > > ntDocumentMapper.java: > > > > > > > > > > 1. at > > > > > > > > > > > org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(Inte > > > > > > rnalIndexShard.java: > > > > > > > > > > 1. at > > > > > > > > > > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary > > > > > > (TransportIndexAction.java: > > > > > > > > > > 1. at > > > > > > > > > > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary > > > > > > (TransportIndexAction.java: > > > > > > > > > > 1. at > > > > > > > > > > > org.elasticsearch.action.support.replication.TransportShardReplicationOpera > > > > > > tionAction > > > > > > > > > > > $AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperat > > > > > > ionAction.java: > > > > > > > > > > 1. at > > > > > > > > > > > org.elasticsearch.action.support.replication.TransportShardReplicationOpera > > > > > > tionAction > > > > > > > > > > $AsyncShardOperationAction.access > > > > > > $400(TransportShardReplicationOperationAction.java:212) > > > > > > at > > > > > > > > > > > org.elasticsearch.action.support.replication.TransportShardReplicationOpera > > > > > > tionAction > > > > > > > > > > $AsyncShardOperationAction > > > > > > $1.run(TransportShardReplicationOperationAction.java:282) > > > > > > at java.util.concurrent.ThreadPoolExecutor > > > > > > $Worker.runTask(ThreadPoolExecutor.java:886) > > > > > > at java.util.concurrent.ThreadPoolExecutor > > > > > > $Worker.run(ThreadPoolExecutor.java:908) > > > > > > at java.lang.Thread.run(Thread.java:619) > > > > > > [2010-10-05 21:01:10,418][DEBUG][action.index] > > > > > > [Abominatrix] [index][3], > > > > > > node[7cfee996-5fc0-4048-8137-b812ac0c7c5f], > > > > > > [P], s[STARTED]: Failed to execute [index {[index][type] > > > > > > [2627f2c1-9c79-4727-8154-99895d8933d2], source[? execute > > > > > > ]}] > > > > > > org.elasticsearch.ElasticSearchParseException: Failed to derive > > > > > > xcontent from [-128, 1, 0, 1, 0, 0, 0, 7, 101, 120, 101, 99, 117, > > > > > > 116, > > > > > > 101, 0, 0, 0, 0, 12, 0, 1, 8, 0, 1, 0, 0, 0, 2, 11, 0, 2, 0, 0, 0, > > > > > > 11, > > > > > > 47, 105, 110, 100, 101, 120, 47, 116, 121, 112, 101, 13, 0, 3, 11, > > > > > > 11, > > > > > > 0, 0, 0, 1, 0, 0, 0, 6, 112, 114, 101, 116, 116, 121, 0, 0, 0, 4, > > > > > > 116, > > > > > > 114, 117, 101, 13, 0, 4, 11, 11, 0, 0, 0, 1, 0, 0, 0, 12, 67, 111, > > > > > > 110, 116, 101, 110, 116, 45, 84, 121, 112, 101, 0, 0, 0, 16, 97, > > > > > > 112, > > > > > > 112, 108, 105, 99, 97, 116, 105, 111, 110, 47, 106, 115, 111, 110, > > > > > > 11, > > > > > > 0, 5, 0, 0, 0, 17, 123, 34, 110, 97, 109, 101, 34, 58, 32, 34, > > > > > > 118, > > > > > > 97, 108, 117, 101, 34, 125, 0, 0] > > > > > > at > > > > > > > > > > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory. > > > > > > java: > > > > > > > > > > 1. at > > > > > > > > > > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory. > > > > > > java: > > > > > > > > > > 1. at > > > > > > > > > > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XConte > > > > > > ntDocumentMapper.java: > > > > > > > > > > 1. at > > > > > > > > > > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XConte > > > > > > ntDocumentMapper.java: > > > > > > > > > > 1. at > > > > > > > > > > > org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(Inte > > > > > > rnalIndexShard.java: > > > > > > > > > > 1. at > > > > > > ... > > > > > > 阅读更多 \>\> ---
**Author:** ![medcl\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/medcl_2/32/3297_2.png) [@medcl\_2](https://discuss.elastic.co/u/medcl_2)\ **Post date:** [October 8, 2010, 1:35pm UTC](https://discuss.elastic.co/t/json-binary-format/3338/12 "2010-10-08T13:35:41Z")
yea,it really a .net client's bug,and i've made it works,thx kimchy. On 10月7日, 下午7时16分, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > Hi, > > Just an update (to the list) after the debug session on IRC. It seems like > thrift framed transport is problematic. In elasticsearch, you can start it > in four modes (thrift.type): threadpool (not framed), threadpool\_framed > (framed), nonblocking (framed - has to be), hsha (framed, has to be). > > If the client and the server are on different protocol (framed vs. not > framed), then obscure problem happens. > > Its importnat for the client and the server to use the same framed, or non > framed, protocol. The first problem was that the .net client was using > framed transport, and the server (es) was using the default threadpool > (which is not framed). > > Once we moved to .net client to be non framed, it worked. Then, we tried > moving es to be framed (threadpool\_framed), and the client framed, but it > still did not work. Seems like a bug in thrift framing support. Not sure > where, smells like the .NET side. > > If someone is using thrift in other lang, would love to hear if it works > in framed mode for them (make sure its an API that uses the body, like > index). > > Thanks, > shay.banon > > 2010/10/7 Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) > > > Great, wanted to check that both server and client are on 0.4 (things have > > changed in thrift). Will catch up on IRC. > > > 2010/10/7 Medcl [me...@163.com](mailto:me...@163.com) > > > sorry for my later reply,and my thrift version is 0.4.0,i use the > > > > > manual build version(0.12.0 snapshot) ,does this effects the data that > > > send to elasticsearch? when will you be the IRC channel? i will on the > > > channel when i am online 🙂 > > > > On 10月6日, 下午10时39分, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > > > > > ok, it looks like within the bytes on elasticsearch side, there is some > > > > data > > > > before the actual content. The big question is where its coming from > > > > (the > > > > exception is thrown during the indexing of the documents). Is there a > > > > chance > > > > that you can get on IRC in a few hours, and I will build some debug > > > > version > > > > of elasticsearch to see where its coming from? > > > > > Which version of thrift do you use? > > > > > -shay.banon > > > > > 2010/10/6 Medcl [me...@163.com](mailto:me...@163.com) > > > > > > the response's status is INTERNAL\_SERVER\_ERROR > > > > > and here is the body's bytearray > > > > > restRequest.Body = > > > > > System.Text.UTF8Encoding.Default.GetBytes("{"a":"b" }"); > > > > > > - > > > > > > > > > > ``` > > > > > restRequest.Body > > > > > {byte[10]} byte[] > > > > > [0] 123 byte > > > > > [1] 34 byte > > > > > [2] 97 byte > > > > > [3] 34 byte > > > > > [4] 58 byte > > > > > [5] 34 byte > > > > > [6] 98 byte > > > > > [7] 34 byte > > > > > [8] 32 byte > > > > > [9] 125 byte > > > > > > > > > > ``` > > > > > > > > > > these bytes can be back to my string（{"a":"b"}） by use > > > > > System.Text.UTF8Encoding.Default.GetString(restRequest.Body); > > > > > > var str = System.Text.UTF8Encoding.Default.GetString(response.Body); > > > > > and the response's body is: > > > > > str: > > > > > { > > > > > "error" : "ReplicationShardOperationFailedException[[index][3] ]; > > > > > nested: ElasticSearchParseException[Failed to derive xcontent from > > > > > [-128, 1, 0, 1, 0, 0, 0, 7, 101, 120, 101, 99, 117, 116, 101, 0, 0, 0, > > > > > 0, 12, 0, 1, 8, 0, 1, 0, 0, 0, 2, 11, 0, 2, 0, 0, 0, 11, 47, 105, 110, > > > > > 100, 101, 120, 47, 116, 121, 112, 101, 13, 0, 3, 11, 11, 0, 0, 0, 1, > > > > > 0, 0, 0, 6, 112, 114, 101, 116, 116, 121, 0, 0, 0, 4, 116, 114, 117, > > > > > 101, 13, 0, 4, 11, 11, 0, 0, 0, 1, 0, 0, 0, 12, 67, 111, 110, 116, > > > > > 101, 110, 116, 45, 84, 121, 112, 101, 0, 0, 0, 16, 97, 112, 112, 108, > > > > > 105, 99, 97, 116, 105, 111, 110, 47, 106, 115, 111, 110, 11, 0, 5, 0, > > > > > 0, 0, 10, 123, 34, 97, 34, 58, 34, 98, 34, 32, 125, 0, 0]]; " > > > > > } > > > > > > On 10月5日, 下午9时49分, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > > > > > > > > > Hey, > > > > > > > Thats really strange, seems like the binary payload when trying > > > > > > to > > > > > > index > > > > > > is not what you sent. There can be several places where this might > > > > > > get > > > > > > messed up, both in elasticsearch and "outside" of it. Can we start > > > > > > with > > > > > > the > > > > > > basics? Can you print/gist the bytes generated (restRequest.Body) > > > > > > before > > > > > > you > > > > > > send the request? > > > > > > > -shay.banon > > > > > > > 2010/10/5 Medcl [me...@163.com](mailto:me...@163.com) > > > > > > > > logs： > > > > > > > [2010-10-05 20:59:10,094][DEBUG][action.index] > > > > > > > [Abominatrix] [index][1], > > > > > > > node[7cfee996-5fc0-4048-8137-b812ac0c7c5f], > > > > > > > [P], s[STARTED]: Failed to execute [index {[index][type][821ce2ad- > > > > > > > e20f-4d59-ba77-f75c820dd94e], source[? execute]}] > > > > > > > org.elasticsearch.ElasticSearchParseException: Failed to derive > > > > > > > xcontent from [-128, 1, 0, 1, 0, 0, 0, 7, 101, 120, 101, 99, 117, > > > > > > > 116, > > > > > > > 101, 0, 0, 0, 0, 12, 0, 1, 8, 0, 1, 0, 0, 0, 2, 11, 0, 2, 0, 0, 0, > > > > > > > 11, > > > > > > > 47, 105, 110, 100, 101, 120, 47, 116, 121, 112, 101, 13, 0, 3, 11, > > > > > > > 11, > > > > > > > 0, 0, 0, 1, 0, 0, 0, 6, 112, 114, 101, 116, 116, 121, 0, 0, 0, 4, > > > > > > > 116, > > > > > > > 114, 117, 101, 13, 0, 4, 11, 11, 0, 0, 0, 1, 0, 0, 0, 12, 67, 111, > > > > > > > 110, 116, 101, 110, 116, 45, 84, 121, 112, 101, 0, 0, 0, 16, 97, > > > > > > > 112, > > > > > > > 112, 108, 105, 99, 97, 116, 105, 111, 110, 47, 120, 115, 111, 110, > > > > > > > 11, > > > > > > > 0, 5, 0, 0, 0, 17, 123, 34, 110, 97, 109, 101, 34, 58, 32, 34, > > > > > > > 118, > > > > > > > 97, 108, 117, 101, 34, 125, 0, 0] > > > > > > > at > > > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory. > > > > > > > > java: > > > > > > > > > > > > 1. at > > > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory. > > > > > > > > java: > > > > > > > > > > > > 1. at > > > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XConte > > > > > > > > ntDocumentMapper.java: > > > > > > > > > > > > 1. at > > > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XConte > > > > > > > > ntDocumentMapper.java: > > > > > > > > > > > > 1. at > > > > org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(Inte > > > > > > > > rnalIndexShard.java: > > > > > > > > > > > > 1. at > > > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary > > > > > > > > (TransportIndexAction.java: > > > > > > > > > > > > 1. at > > > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary > > > > > > > > (TransportIndexAction.java: > > > > > > > > > > > > 1. at > > > > org.elasticsearch.action.support.replication.TransportShardReplicationOpera > > > > > > > > tionAction > > > > $AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperat > > > > > > > > ionAction.java: > > > > > > > > > > > > 1. at > > > > org.elasticsearch.action.support.replication.TransportShardReplicationOpera > > > > > > > > tionAction > > > > > > > > > > > > $AsyncShardOperationAction.access > > > > > > > $400(TransportShardReplicationOperationAction.java:212) > > > > > > > at > > > > org.elasticsearch.action.support.replication.TransportShardReplicationOpera > > > > > > > > tionAction > > > > > > > > > > > > $AsyncShardOperationAction > > > > > > > $1.run(TransportShardReplicationOperationAction.java:282) > > > > > > > at java.util.concurrent.ThreadPoolExecutor > > > > > > > $Worker.runTask(ThreadPoolExecutor.java:886) > > > > > > > at java.util.concurrent.ThreadPoolExecutor > > > > > > > $Worker.run(ThreadPoolExecutor.java:908) > > > > > > > at java.lang.Thread.run(Thread.java:619) > > > > > > > [2010-10-05 20:59:59,384][DEBUG][action.index] > > > > > > > [Abominatrix] [index][0], > > > > > > > node[7cfee996-5fc0-4048-8137-b812ac0c7c5f], > > > > > > > [P], s[STARTED]: Failed to execute [index {[index][type] > > > > > > > [1b080d59-7b60-4b98-97ba-e1b9b492a5e9], source[? execute > > > > > > > ]}] > > > > > > > org.elasticsearch.ElasticSearchParseException: Failed to derive > > > > > > > xcontent from [-128, 1, 0, 1, 0, 0, 0, 7, 101, 120, 101, 99, 117, > > > > > > > 116, > > > > > > > 101, 0, 0, 0, 0, 12, 0, 1, 8, 0, 1, 0, 0, 0, 2, 11, 0, 2, 0, 0, 0, > > > > > > > 11, > > > > > > > 47, 105, 110, 100, 101, 120, 47, 116, 121, 112, 101, 13, 0, 3, 11, > > > > > > > 11, > > > > > > > 0, 0, 0, 1, 0, 0, 0, 6, 112, 114, 101, 116, 116, 121, 0, 0, 0, 4, > > > > > > > 116, > > > > > > > 114, 117, 101, 13, 0, 4, 11, 11, 0, 0, 0, 1, 0, 0, 0, 12, 67, 111, > > > > > > > 110, 116, 101, 110, 116, 45, 84, 121, 112, 101, 0, 0, 0, 16, 97, > > > > > > > 112, > > > > > > > 112, 108, 105, 99, 97, 116, 105, 111, 110, 47, 120, 115, 111, 110, > > > > > > > 11, > > > > > > > 0, 5, 0, 0, 0, 17, 123, 34, 110, 97, 109, 101, 34, 58, 32, 34, > > > > > > > 118, > > > > > > > 97, 108, 117, 101, 34, 125, 0, 0] > > > > > > > at > > > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory. > > > > > > > > java: > > > > > > > > > > > > 1. at > > > > org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory. > > > > > > > > java: > > > > > > > > > > > > 1. at > > > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XConte > > > > > > > > ntDocumentMapper.java: > > > > > > > > > > > > 1. at > > > > org.elasticsearch.index.mapper.xcontent.XContentDocumentMapper.parse(XConte > > > > > > > > ntDocumentMapper.java: > > > > > > > > > > > > 1. at > > > > org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(Inte > > > > > > > > rnalIndexShard.java: > > > > > > > > > > > > 1. at > > > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary > > > > > > > > (TransportIndexAction.java: > > > > > > > > > > > > 1. at > > > > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary > > > > > > > > (TransportIndexAction.java: > > > > > > > > > > > > 1. at > > > > org.elasticsearch.action.support.replication.TransportShardReplicationOpera > > ... > > 阅读更多 \>\> ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:18am UTC](https://discuss.elastic.co/t/json-binary-format/3338/13 "2017-07-06T04:18:13Z")