# Streaming a whole SearchResponse as JSON **URL:** \ **Category:** Elasticsearch\ **Created:** [December 21, 2010, 11:09pm UTC](https://discuss.elastic.co/t/streaming-a-whole-searchresponse-as-json/3688 "2010-12-21T23:09:36Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![skaffman\_2](https://avatars.discourse-cdn.com/v4/letter/s/c4cdca/32.png) [@skaffman\_2](https://discuss.elastic.co/u/skaffman_2)\ **Post date:** [December 21, 2010, 11:09pm UTC](https://discuss.elastic.co/t/streaming-a-whole-searchresponse-as-json/3688/1 "2010-12-21T23:09:36Z")
I'm writing a thin proxy server to sit in front of ElasticSearch, using the java client API. The proxy constructs a query, sends it off to ElasticSearch, and returns the results to the proxy client. Pretty simple, but I'm struggling with how to take the SearchResponse and get it to stream itself as JSON to the proxy's HTTP response. The ToXContent and Streamable interfaces seem to only work with low-level data, not JSON. The cloest I can get to is ``` searchResponse.getHits().writeTo(new DataOutputStreamOutput(new ``` DataOutputStream(httpResponse.getOutputStream()))); ... but of course, this writes gibberish to the stream, nut human- readable JSON. Am I looking in the wrong place? I'm considering using the REST client inside the proxy instead, but it seems a shame to forgo the extra performance of the Java Node client, given that the proxy is written in Java. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [December 21, 2010, 11:23pm UTC](https://discuss.elastic.co/t/streaming-a-whole-searchresponse-as-json/3688/2 "2010-12-21T23:23:00Z")
You can get an XContentBuilder using XContentFactory.jsonBuilder(), and then use SearchResponse to generate the result in json, for example: XContetBuilder builder = XContentFactory.jsonBuilder(); builder.startObject(); searchResponse.toXContent(builder, ToXContent.EMPTY\_PARAMS); builder.endObject(); from the builder you can get the string, or bytes, with the json response. -shay.banon On Wed, Dec 22, 2010 at 1:09 AM, skaffman [kenny@chasmcity.com](mailto:kenny@chasmcity.com) wrote: > I'm writing a thin proxy server to sit in front of Elasticsearch, > using the java client API. The proxy constructs a query, sends it off > to Elasticsearch, and returns the results to the proxy client. > > Pretty simple, but I'm struggling with how to take the SearchResponse > and get it to stream itself as JSON to the proxy's HTTP response. The > ToXContent and Streamable interfaces seem to only work with low-level > data, not JSON. The cloest I can get to is > > searchResponse.getHits().writeTo(new DataOutputStreamOutput(new > DataOutputStream(httpResponse.getOutputStream()))); > > ... but of course, this writes gibberish to the stream, nut human- > readable JSON. > > Am I looking in the wrong place? > > I'm considering using the REST client inside the proxy instead, but it > seems a shame to forgo the extra performance of the Java Node client, > given that the proxy is written in Java. ---
**Author:** ![skaffman\_2](https://avatars.discourse-cdn.com/v4/letter/s/c4cdca/32.png) [@skaffman\_2](https://discuss.elastic.co/u/skaffman_2)\ **Post date:** [December 22, 2010, 10:43am UTC](https://discuss.elastic.co/t/streaming-a-whole-searchresponse-as-json/3688/3 "2010-12-22T10:43:42Z")
Ah cool, that works nicely. It does, however, mean shifting one in-memory representation to another before streaming it out, which could be a bottleneck. It would nice if it could be streamed out directly. On Dec 21, 11:23 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > You can get an XContentBuilder using XContentFactory.jsonBuilder(), and then > use SearchResponse to generate the result in json, for example: > > XContetBuilder builder = XContentFactory.jsonBuilder(); > builder.startObject(); > searchResponse.toXContent(builder, ToXContent.EMPTY\_PARAMS); > builder.endObject(); > > from the builder you can get the string, or bytes, with the json response. > > -shay.banon > > On Wed, Dec 22, 2010 at 1:09 AM, skaffman [ke...@chasmcity.com](mailto:ke...@chasmcity.com) wrote: > > > I'm writing a thin proxy server to sit in front of Elasticsearch, > > using the java client API. The proxy constructs a query, sends it off > > to Elasticsearch, and returns the results to the proxy client. > > > Pretty simple, but I'm struggling with how to take the SearchResponse > > and get it to stream itself as JSON to the proxy's HTTP response. The > > ToXContent and Streamable interfaces seem to only work with low-level > > data, not JSON. The cloest I can get to is > > > searchResponse.getHits().writeTo(new DataOutputStreamOutput(new > > DataOutputStream(httpResponse.getOutputStream()))); > > > ... but of course, this writes gibberish to the stream, nut human- > > readable JSON. > > > Am I looking in the wrong place? > > > I'm considering using the REST client inside the proxy instead, but it > > seems a shame to forgo the extra performance of the Java Node client, > > given that the proxy is written in Java. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [December 22, 2010, 11:25am UTC](https://discuss.elastic.co/t/streaming-a-whole-searchresponse-as-json/3688/4 "2010-12-22T11:25:50Z")
I'll add a constructor to XContentBuilder, so you can do: XContentBuilder(XContent xContent, OutputStream bos). Just make sure to call close on the builder when you are done with it. in ES, I generate the bytes into thread local byte buffer, and then reuse it on the networking layer. Thats why the unsafeBytes / unsafeBytesLength are there on the builder. Don't know which network lib you use..., but the OutputStream option should be good on servlet envs. -shay.banon On Wed, Dec 22, 2010 at 12:43 PM, skaffman [kenny@chasmcity.com](mailto:kenny@chasmcity.com) wrote: > Ah cool, that works nicely. > > It does, however, mean shifting one in-memory representation to > another before streaming it out, which could be a bottleneck. It would > nice if it could be streamed out directly. > > On Dec 21, 11:23 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > You can get an XContentBuilder using XContentFactory.jsonBuilder(), and > > then > > use SearchResponse to generate the result in json, for example: > > > > XContetBuilder builder = XContentFactory.jsonBuilder(); > > builder.startObject(); > > searchResponse.toXContent(builder, ToXContent.EMPTY\_PARAMS); > > builder.endObject(); > > > > from the builder you can get the string, or bytes, with the json > > response. > > > > -shay.banon > > > > On Wed, Dec 22, 2010 at 1:09 AM, skaffman [ke...@chasmcity.com](mailto:ke...@chasmcity.com) wrote: > > > > > I'm writing a thin proxy server to sit in front of Elasticsearch, > > > using the java client API. The proxy constructs a query, sends it off > > > to Elasticsearch, and returns the results to the proxy client. > > > > > Pretty simple, but I'm struggling with how to take the SearchResponse > > > and get it to stream itself as JSON to the proxy's HTTP response. The > > > ToXContent and Streamable interfaces seem to only work with low-level > > > data, not JSON. The cloest I can get to is > > > > > searchResponse.getHits().writeTo(new DataOutputStreamOutput(new > > > DataOutputStream(httpResponse.getOutputStream()))); > > > > > ... but of course, this writes gibberish to the stream, nut human- > > > readable JSON. > > > > > Am I looking in the wrong place? > > > > > I'm considering using the REST client inside the proxy instead, but it > > > seems a shame to forgo the extra performance of the Java Node client, > > > given that the proxy is written in Java. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:15am UTC](https://discuss.elastic.co/t/streaming-a-whole-searchresponse-as-json/3688/5 "2017-07-06T04:15:00Z")