SearchResponse Vs CURL response. Aggregations responses don't match

[Shankar_Daruga]

Hello,

In our current configuration, with 5.2 jar, the 'SearchResponse' Object, after converting it to JSON using XContentBuilder, the JSON matches that of the CURL request.

We are now trying to upgrade to 6.6 and so trying to integrate the 6.6 jar which more or less has the searchResponse object already.

Now, the searchResponse object which is giving a different output for aggregation responses. Please have a look at the examples below.

Let us take a sample example

{
  "aggregations": {
    "nested_level1": {
      "nested": {
        "path": "levels.level1"
      },
      "aggregations": {
        "filter_block": {
          "filter": {
            "bool": {
              "must": [
                {
                  "term": {
                    "levels.level1.name": {
                      "value": "elastic"
                    }
                  }
                }
              ]
            },
            "aggregations": {
              "phoneNumber": {
                "terms": {
                  "field": "levels.level1.phone",
                  "size": 2000
                }
              }
            }
          }
        }
      }
    }
  }
}

The response that I'm getting from the SearchResponse object is something like below:

{
  "nested#nested_level1": {
    "doc_count": 5,
    "filter#filter_block": {
      "doc_count": 4,
      "sterms#phoneNumber": {
        "doc_count_error_upper_bound": 0,
        "sum_other_doc_count": 0,
        "buckets": [
          {
            "key": "Smart Homes",
            "doc_count": 1,
            "reverse_nested#reverse_nested_to_base": {
              "doc_count": 1
            }
          }
        ]
      }
    }
  }
}

Why the names in aggregation request are being modified by prefixing the block type?. Is there a way to convert the searchResponse Object received to JSON string which would be identical to the CURL response in 6.6 jar?

Appreciate you help in this regard.

[dadoonet (David Pilato)]

Why the names in aggregation request are being modified by prefixing the block type?

That's the only way to convert back the JSON to the right aggregation response object.

Is there a way to convert the searchResponse Object received to JSON string which would be identical to the CURL response in 6.6 jar?

I don't think so.
But if you are using Java, why not using the Java objects instead of a JSON?

[Shankar_Daruga]

Well our case is something like below.

We have a custom search request building (build query for ES) and a search response processing code (convert ES to our own custom response) in our service that has been there and serving a lot of clients from a long time.

Currently, it uses the 'SearchRequestBuilder' something like below:

SearchRequestBuilder srb
//set all the attributes
SearchResponse searchResponse = srb?.setTimeout(TimeValue.timeValueSeconds(90))?.execute()?.actionGet()

Now the 'searchResponse' above when converted to JSON would give the same result as in CURL request for aggregations. (elasticsearch:5.2.0)

Although we've used the QueryBuilder and client jar's other classes when building the query, we've decoupled our search response's processing code from the client's classes for our own benefit so that there is not need to fiddle upgrade/modify our custom processing code to integrate with the new jar changes. It was based on assumption that client jar is more prone to change frequently over time than the standardized and probably widely used simple JSON response.

Please note that we have our own methods to match the aggs request with JSON response in case you are wondering.

Yet, it would've been good to get the actual JSON response somehow as it makes our integration job much eaiser. If there is none, thats okay and seems that our initial assumption didn't work out good.

Thanks for your help and support. Really appreciate it.

[dadoonet (David Pilato)]

Did you try the REST Client instead ?

At least you can always generate a low level call which is basically like a curl and just get the whole response as a JSon document.

[Shankar_Daruga]

Yes, thank you. Low level client works for our case. However, we've made a decision to go with the REST High Level Client, the reason being, we also use our clients in indexing side which needs support to create indices, update mappings and bulk index the documents with minimal changes. Aso, we don't want to marshall the search request ourselves which could potentially contain some values that may break the simple toJSON conversion methods.

[dadoonet (David Pilato)]

I'm doing the same.
Using the HLCLient anytime I can. Using the LLClient when a feature is not implemented yet.

You can just call:

client.getLowLevelClient().performRequest(...)

[Shankar_Daruga]

Oh yes. This is very useful. Use HighLevelClient where ever possible and jump back to lowLevel when you absolutely need it. Thank you.