# Is JSON too verbose for high performance **URL:** \ **Category:** Elasticsearch\ **Created:** [November 2, 2015, 9:27pm UTC](https://discuss.elastic.co/t/is-json-too-verbose-for-high-performance/33570 "2015-11-02T21:27:18Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![iamthealex](https://avatars.discourse-cdn.com/v4/letter/i/e9c0ed/32.png) [@iamthealex](https://discuss.elastic.co/u/iamthealex)\ **Post date:** [November 2, 2015, 9:27pm UTC](https://discuss.elastic.co/t/is-json-too-verbose-for-high-performance/33570/1 "2015-11-02T21:27:18Z")
JSON is quite verbose. Google introduced protocol buffers many years ago, and one of the the key features was to reduce the number of bytes that need to go "over the wire". Newer attempts to optimize for bigdata use formats like those available in parquet (parquet.apache.org). There are even now thrift bindings for parquet. It seems a step backwards to go back to json. Am I missing something basic here? I suppose that I could push (using a Kafka publisher method) to a server (that subscribes to the relevant Kafka topic) using a compressed format, and then do the expansion to json just before inserting into elasticsearch. Can you comment on the best way to take data from servers distributed worldwide and get the data into elasticsearch? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 3, 2015, 6:59am UTC](https://discuss.elastic.co/t/is-json-too-verbose-for-high-performance/33570/2 "2015-11-03T06:59:32Z")
My question back to you would be, are you having issues now, and what does your setup look like? ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [November 3, 2015, 8:46am UTC](https://discuss.elastic.co/t/is-json-too-verbose-for-high-performance/33570/3 "2015-11-03T08:46:16Z")
You are invited to revive the stopped thrift plugin [https://github.com/elastic/elasticsearch-transport-thrift](https://github.com/elastic/elasticsearch-transport-thrift) and demonstrate how to gain more performance over JSON. Some notes: - JSON is only used for HTTP API - ES uses a fast builtin binary protocol between nodes which is not JSON but already binary encoded and compressed - ES/Lucene uses a lot of compression in the storage backend to speed up I/O which dominates the overall performance, there is no JSON on this system level - ES offers alternatives, CBOR [https://tools.ietf.org/html/rfc7049](https://tools.ietf.org/html/rfc7049) and SMILE [http://wiki.fasterxml.com/SmileFormat](http://wiki.fasterxml.com/SmileFormat) to store cluster state - the possible performance gain by switching a serialization format must be balanced to the readability of the data transmitted by client apps and users which are used to a data representation language like JSON ---
**Author:** ![mgrigorov](https://avatars.discourse-cdn.com/v4/letter/m/43a26b/32.png) [@mgrigorov](https://discuss.elastic.co/u/mgrigorov)\ **Post date:** [November 3, 2015, 1:05pm UTC](https://discuss.elastic.co/t/is-json-too-verbose-for-high-performance/33570/4 "2015-11-03T13:05:03Z")
Hi, On Tue, Nov 3, 2015 at 9:56 AM, Jörg Prante [noreply@discuss.elastic.co](mailto:noreply@discuss.elastic.co) wrote: > jprante [http://discuss.elastic.co/users/jprante](http://discuss.elastic.co/users/jprante) Jörg Prante > [http://discuss.elastic.co/users/jprante](http://discuss.elastic.co/users/jprante) > November 3 > > You are invited to revive the stopped thrift plugin > [GitHub - elastic/elasticsearch-transport-thrift: Thrift Transport for elasticsearch (STOPPED)](https://github.com/elastic/elasticsearch-transport-thrift) and demonstrate > how to gain more performance over JSON. > > Some notes: > > - JSON is only used for HTTP API > - ES uses a fast builtin binary protocol between nodes which is not > JSON but already binary encoded and compressed > > +1! > - ES/Lucene uses a lot of compression in the storage backend to speed > up I/O which dominates the overall performance, there is no JSON on this > system level > - ES offers alternatives, CBOR [RFC 7049 - Concise Binary Object Representation (CBOR)](https://tools.ietf.org/html/rfc7049) and > SMILE [http://wiki.fasterxml.com/SmileFormat](http://wiki.fasterxml.com/SmileFormat) to store cluster state > - the possible performance gain by switching a serialization format > must be balanced to the readability of the data transmitted by client apps > and users which are used to a data representation language like JSON > > I think the best solution would be use content negotiation. The client may > send request header "Accept" with the list of supported/preferred response > content types. If they are supported (even by plugin) then serve the best > match, otherwise fallback to JSON. ---
**Author:** ![iamthealex](https://avatars.discourse-cdn.com/v4/letter/i/e9c0ed/32.png) [@iamthealex](https://discuss.elastic.co/u/iamthealex)\ **Post date:** [November 3, 2015, 2:07pm UTC](https://discuss.elastic.co/t/is-json-too-verbose-for-high-performance/33570/5 "2015-11-03T14:07:24Z")
Thanks for all the responses. @warkolm I am not having issues now. I'm trying to select a set of tools to use for our new project. @jprante I particularly like the idea of content negotiation with Accept headers. This approach will allow me start out with JSON and to migrate to some more compressed format if the need arises. @mgrigorov If the need arises for something more compact than JSON, I will consider reviving the stopped thrift plugin. Was is abandoned because there was no measurable performance improvement? ---
**Author:** ![jillesvangurp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jillesvangurp/32/3863_2.png) [@jillesvangurp](https://discuss.elastic.co/u/jillesvangurp)\ **Post date:** [November 3, 2015, 4:01pm UTC](https://discuss.elastic.co/t/is-json-too-verbose-for-high-performance/33570/6 "2015-11-03T16:01:06Z")
There's a binary version of json that is supported by mongodb called BSON. There's a add on project for jackson that adds BSON support to jackson and this allows you to easily parse and serialize to BSON: [https://github.com/michel-kraemer/bson4jackson](https://github.com/michel-kraemer/bson4jackson). I've actually integrated that recently into my jsonj library (which uses jackson). But I don't recommend using it unless you really need it (e.g. to integrate with mongodb). There are a few counter intuitive things about BSON. If size is you main concern: don't use it. You'll actually use up more space typically. If parsing overhead is your main concern, the biggest penalty is mapping to in memory object structures. This doesn't change a lot between bson and json. Jackson includes a streaming JSON parser which is pretty much as fast as it gets (for json). As far as I can measure BSON support in jackson is more verbose and not a whole lot faster. Which, is why recommend not using it. As far as I understand the Elasticsearch architecture, it uses a streaming JSON parser. I don't think the performance would differ hugely if they were to use BSON (might actually degrade a little). Much of the benefit from dedicated binary protocols comes from their reduced size. You can probably get similar improvements from simply enable gzip compression (which es supports). However binary protocols are difficult to deal with in terms of evolving the API, adding new features and using the API over HTTP. Elasticsearch avoids a lot of parsing and serializing overhead. For example, it doesn't reconstruct in memory trees for your documents in most cases and the way it fetches and stores document json is pretty efficient already. Also, given that it is actually a json document store, it would be kind of weird to not have a json based API. It is true that parser overhead is typically lower for binary protocols. However, parsing overhead is unlikely a big factor in overal ES performance for most setups. This is why e.g. Logstash now recommends using the http protocol as of 2.0. The advantage of using an embedded node and its internal binary protocol is just not enough to justify the complexity of that solution. ---
**Author:** ![iamthealex](https://avatars.discourse-cdn.com/v4/letter/i/e9c0ed/32.png) [@iamthealex](https://discuss.elastic.co/u/iamthealex)\ **Post date:** [November 13, 2015, 9:07pm UTC](https://discuss.elastic.co/t/is-json-too-verbose-for-high-performance/33570/7 "2015-11-13T21:07:43Z")
@jillesvangurp Thanks for all the replies. The project is new and I'm not concerned about performance yet. I did appreciate the answer I got; I now understand better what is going on "underneath the covers". ---
**Author:** ![taowen](https://avatars.discourse-cdn.com/v4/letter/t/6f9a4e/32.png) [@taowen](https://discuss.elastic.co/u/taowen)\ **Post date:** [January 17, 2017, 2:20am UTC](https://discuss.elastic.co/t/is-json-too-verbose-for-high-performance/33570/8 "2017-01-17T02:20:26Z")
thrift is not necessarily faster. JSON is not that slow either. [https://www.codeproject.com/Articles/1165627/Jsoniter-JSON-is-faster-than-thrift-avro](https://www.codeproject.com/Articles/1165627/Jsoniter-JSON-is-faster-than-thrift-avro) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:04pm UTC](https://discuss.elastic.co/t/is-json-too-verbose-for-high-performance/33570/9 "2017-07-05T22:04:08Z")