SearchResponse as JSON

Elastic Stack Elasticsearch
1

Hi, I'm using version 8.5.1 w/ the Java client.

I'm looking for a way to get the SearchResponse as JSON, or at least the same output I'd get when executing the query in Kibana.

Thanks for any hints.

2

This is how I'm doing this:

This should be available in the future with:

3

Thank you for your insights.

It's almost what I want. The JSON differs to the response I get via the REST API though.

here's a snippet of the response:

"sum#rxPackets": {
  "value": 1.8250254996E10
},

compared to

"rxBytes": {
  "value": 18250254996
},

with the REST API I'd get the long values, here I'd have to convert all the values. also the aggregation type is added, see snippets above.

Is there any way to have the JSON the same way as with the REST API response?

Thanks

4

@swallez Do you know? Is there a way to get at least the same "value"?

5

that would be great!

I'm experiencing strange things.

For now I tried using Double.valueOf(valueAsText).longValue() .

But I realised that this is not correct. Up front, when running the same query in Kibana I don't have any problems, the results are fine.

I have to strike out a bit.

I calculate a hash (long) value of a unique string (serial number of a device, which can be anything in A-Z0-9), and index the hash as long value. I use this hash as field for a histogram aggregation.

When processing the result I try to lookup the hash - and I'm not able to find a result.

The value I index: -4872811286368342000
The value I parse: -4872811286368342016, from text: -4.872811286368342E18 (this is what I get from the SearchResponse)

"aggs": {
  "devices": {
    "histogram": {
      "field": "serialHash",
      "interval": 1,
      "min_doc_count": 1
    },
    ...
}

Any idea would be much appreciated. Thanks

6

You can remove the aggregation type from the serialized string using these instructions: Serializing aggregations and suggestions without typed keys | Elasticsearch Java API Client [8.5] | Elastic

Regarding the sum aggregation result, the Java client assumes this aggregation's result is a double.

7

Ok thanks.

and regarding the histogram aggregation, do you have any ideas?

the indexed field is of type long, but the result also seems to be assumed double, or at least I get the exponential writing. The result in Kibana is correct though.

8

Can you expand on the long issue in histograms? Where does valueAsText come from in Double.valueOf(valueAsText).longValue()?

9

Of course.
I index the value as long, and do a histogram on that field, and what I get back in the SearchResponse is

"doc_count": 3600,
"key": -1.49437614230614451E18

getting the value from the key from the SearchResponse as JSON asText or asDouble gives me the same long value, but the value is not the same as I index.

When querying via Kibana I get the correct long value as key for my histogram.

I don't understand why I get a different value with Java than with Kibana. I would have hoped that when I do a histogram on long values the same value is put into the response as key.

I hope I was able to describe my problem properly.
Thanks

Edit

I just double checked the SearchResponse object before converting it to JSON, and at that point I already have the double values for the keys in question, so even for a histogram on long values the Java API gives me a double value as key, and this one is not the same as I index

What I quite don't understand is why I get the correct values when using Kibana.

And why is even the sum aggregation handled differently? (but that's just a minor problem/concern

10

I wonder if this is a result of limits to JSON precision and differences in the libraries used to create/parse the JSON?

11

It's already in the SearchResponse before anything gets converted. and what I get with the java-api, and, the result w/ Kibana is fine.
I'm really curious why the key of the histogram (which is of type long), is converted to double, or treated as double, before sending it back to the client.
a double would still be fine, if the conversion to long would be the same value as the original.

12

Can you provide a code snippet showing how you do the number conversion? The precision loss may happen there.

Histogram keys are computed as double values in the Elasticsearch server, i.e. they cannot represent integer values larger than 2^53 without precision loss. In your case -4872811286368342000 uses 63 bits. If we consider this number without the trailing 000, then -4872811286368342 fits in 53 bits, which is the limit of the precision loss. So depending on how the JSON library formats/parses this value and converts it to a long (i.e. extends with leading zeros or uses base-2 arithmetic) we may have different results.

That is to say that although we may find a solution for this particular case, using large integer values in JSON is prone to this kind of rounding errors, depending on how the JSON is created/processed.

1 Like
13

well, this is what I do:

I get the SearchResponse from client.search(...), this result I convert to Json as David showed me above:

With Jackson library I read the nodes, and get they value for the histogram key via Double d = node.get("key").asDouble(); and then I do d.longValue() (standard Java).

I also tried

String keyAsText = node.get("key").asText(); 
Double.valueOf(keyAsText).longValue()

that's all I do. But there. must be something different than you do when you return the result via REST? as Kibana shows the corret long value.

Please let me know if you need anything else.

14

Why do you serialize the response to a string, and then parse it into a JSON tree to get the information you need? What prevents you from using the response object directly?

Something like response.aggregations().get("devices").histogram().buckets().array().get(0).key()

15

the reason is just b/c I forward the result to another service which handles the JSON result and doesn't know about ES specific classes.

and even with your suggestion it's a double what I get from the SearchResponse, so I'm stuck there too

16

I think I'll update my mappings and use int for my hash.

I thought with histograms on a field w/ a numeric type there is no calculation needed for the key, as I thought the key is just the value of the field itself. I guess this is wrong?

Still what confuses me is that I get the correct values in Kibana! Do you have any explanation why?

17

Ok, I understand now. Here's a way to get the response result in its raw binary form. It uses some lower level abstractions of the Java client, namely the transport that processes http requests and response, and endpoints that define, for each API endpoint, how to serialize requests and deserialize responses.

The trick is to wrap the _search endpoint into an endpoint that returns a raw binary response.

We will add this as an out-of-the-box feature of the Java client in a future release.

// Create the search request
SearchRequest request = SearchRequest.of(b -> b...);

// Create a binary endpoint from the regular search endpoint. It will not deserialize
// the response and instead will just return the raw response input stream.
SimpleEndpoint<SearchRequest, ?> endpoint = SearchRequest._ENDPOINT;

BinaryEndpoint<SearchRequest> binarySearchEndpoint = new BinaryEndpoint<>(
    endpoint.id(),
    endpoint::method,
    endpoint::requestUrl,
    endpoint::queryParameters,
    endpoint::headers,
    endpoint.hasRequestBody(),
    null
);

// Force typed_keys to false, so that aggregations names do not hold type information
TransportOptions options = esClient._transportOptions().toBuilder()
    .setParameter("typed_keys", "false")
    .build();

// Call Elasticsearch by providing the transport the request and endpoint
BinaryResponse binaryResponse = esClient._transport().performRequest(request, binarySearchEndpoint, options);


// Do something with the response
System.out.println("Response = " + IOUtils.toString(binaryResponse.content(), StandardCharsets.UTF_8));
2 Likes
18

Edit: I added transport options with typed_keys=false in the code snippet above.

19

Thanks. this is neat as well.

As the issue w/ the double/long precision issue I know use Murmur3 hashing to get an int, instead of the CityHash long. This now works as expected.

20

Awesome!