Streaming a whole SearchResponse as JSON

Elastic Stack Elasticsearch
1

I'm writing a thin proxy server to sit in front of ElasticSearch,
using the java client API. The proxy constructs a query, sends it off
to ElasticSearch, and returns the results to the proxy client.

Pretty simple, but I'm struggling with how to take the SearchResponse
and get it to stream itself as JSON to the proxy's HTTP response. The
ToXContent and Streamable interfaces seem to only work with low-level
data, not JSON. The cloest I can get to is

searchResponse.getHits().writeTo(new DataOutputStreamOutput(new

DataOutputStream(httpResponse.getOutputStream())));

... but of course, this writes gibberish to the stream, nut human-
readable JSON.

Am I looking in the wrong place?

I'm considering using the REST client inside the proxy instead, but it
seems a shame to forgo the extra performance of the Java Node client,
given that the proxy is written in Java.

2

You can get an XContentBuilder using XContentFactory.jsonBuilder(), and then
use SearchResponse to generate the result in json, for example:

XContetBuilder builder = XContentFactory.jsonBuilder();
builder.startObject();
searchResponse.toXContent(builder, ToXContent.EMPTY_PARAMS);
builder.endObject();

from the builder you can get the string, or bytes, with the json response.

-shay.banon

On Wed, Dec 22, 2010 at 1:09 AM, skaffman kenny@chasmcity.com wrote:

I'm writing a thin proxy server to sit in front of Elasticsearch,
using the java client API. The proxy constructs a query, sends it off
to Elasticsearch, and returns the results to the proxy client.

Pretty simple, but I'm struggling with how to take the SearchResponse
and get it to stream itself as JSON to the proxy's HTTP response. The
ToXContent and Streamable interfaces seem to only work with low-level
data, not JSON. The cloest I can get to is

searchResponse.getHits().writeTo(new DataOutputStreamOutput(new
DataOutputStream(httpResponse.getOutputStream())));

... but of course, this writes gibberish to the stream, nut human-
readable JSON.

Am I looking in the wrong place?

I'm considering using the REST client inside the proxy instead, but it
seems a shame to forgo the extra performance of the Java Node client,
given that the proxy is written in Java.

3

Ah cool, that works nicely.

It does, however, mean shifting one in-memory representation to
another before streaming it out, which could be a bottleneck. It would
nice if it could be streamed out directly.

On Dec 21, 11:23 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

You can get an XContentBuilder using XContentFactory.jsonBuilder(), and then
use SearchResponse to generate the result in json, for example:

XContetBuilder builder = XContentFactory.jsonBuilder();
builder.startObject();
searchResponse.toXContent(builder, ToXContent.EMPTY_PARAMS);
builder.endObject();

from the builder you can get the string, or bytes, with the json response.

-shay.banon

On Wed, Dec 22, 2010 at 1:09 AM, skaffman ke...@chasmcity.com wrote:

I'm writing a thin proxy server to sit in front of Elasticsearch,
using the java client API. The proxy constructs a query, sends it off
to Elasticsearch, and returns the results to the proxy client.

Pretty simple, but I'm struggling with how to take the SearchResponse
and get it to stream itself as JSON to the proxy's HTTP response. The
ToXContent and Streamable interfaces seem to only work with low-level
data, not JSON. The cloest I can get to is

searchResponse.getHits().writeTo(new DataOutputStreamOutput(new
DataOutputStream(httpResponse.getOutputStream())));

... but of course, this writes gibberish to the stream, nut human-
readable JSON.

Am I looking in the wrong place?

I'm considering using the REST client inside the proxy instead, but it
seems a shame to forgo the extra performance of the Java Node client,
given that the proxy is written in Java.

4

I'll add a constructor to XContentBuilder, so you can
do: XContentBuilder(XContent xContent, OutputStream bos). Just make sure to
call close on the builder when you are done with it.

in ES, I generate the bytes into thread local byte buffer, and then reuse it
on the networking layer. Thats why the unsafeBytes / unsafeBytesLength are
there on the builder. Don't know which network lib you use..., but the
OutputStream option should be good on servlet envs.

-shay.banon

On Wed, Dec 22, 2010 at 12:43 PM, skaffman kenny@chasmcity.com wrote:

Ah cool, that works nicely.

It does, however, mean shifting one in-memory representation to
another before streaming it out, which could be a bottleneck. It would
nice if it could be streamed out directly.

On Dec 21, 11:23 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

You can get an XContentBuilder using XContentFactory.jsonBuilder(), and
then
use SearchResponse to generate the result in json, for example:

XContetBuilder builder = XContentFactory.jsonBuilder();
builder.startObject();
searchResponse.toXContent(builder, ToXContent.EMPTY_PARAMS);
builder.endObject();

from the builder you can get the string, or bytes, with the json
response.

-shay.banon

On Wed, Dec 22, 2010 at 1:09 AM, skaffman ke...@chasmcity.com wrote:

I'm writing a thin proxy server to sit in front of Elasticsearch,
using the java client API. The proxy constructs a query, sends it off
to Elasticsearch, and returns the results to the proxy client.

Pretty simple, but I'm struggling with how to take the SearchResponse
and get it to stream itself as JSON to the proxy's HTTP response. The
ToXContent and Streamable interfaces seem to only work with low-level
data, not JSON. The cloest I can get to is

searchResponse.getHits().writeTo(new DataOutputStreamOutput(new
DataOutputStream(httpResponse.getOutputStream())));

... but of course, this writes gibberish to the stream, nut human-
readable JSON.

Am I looking in the wrong place?

I'm considering using the REST client inside the proxy instead, but it
seems a shame to forgo the extra performance of the Java Node client,
given that the proxy is written in Java.