Elasticsearch json response to SearchResponse object

Monte_A · March 17, 2018, 9:20am

Hi There,

Could someone help to get it working?
So far I got the solution below. This works for simple queries, however it doesn't work for aggregations and I am getting exception ParsingException "Could not parse aggregation keyed as [dashboard]"

JsonXContentParser xContentParser = new JsonXContentParser(NamedXContentRegistry.EMPTY,
new JsonFactory().createParser(SearchResponse response = SearchResponse.fromXContent(xContentParser);

The response I need to parse is provided below:
{
"took": 16,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 2,
"max_score": 1,
"hits": [
{
"_index": "demo",
"_type": "demo",
"_id": "OSScKmIBKBDaRuGPyr42",
"_score": 1,
"_source": {
"demoId": "2",
"demoName": "j",
"uniqueDemo": "jab1980xy11b"
}
},
{
"_index": "demo",
"_type": "demo",
"_id": "OCScKmIBKBDaRuGPyr42",
"_score": 1,
"_source": {
"demoId": "2",
"demoName": "j",
"uniqueDemo": "jmb1970oi611234554321123a"
}
}
]
},
"aggregations": {
"dashboard": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "jab1980xy11b",
"doc_count": 1,
"data": {
"hits": {
"total": 1,
"max_score": 1,
"hits": [
{
"_index": "demo",
"_type": "demo",
"_id": "OSScKmIBKBDaRuGPyr42",
"_score": 1,
"_source": {
"demoId": "2",
"demoName": "j"
}
}
]
}
}
},
{
"key": "jmb1970oi611234554321123a",
"doc_count": 1,
"data": {
"hits": {
"total": 1,
"max_score": 1,
"hits": [
{
"_index": "demo",
"_type": "demo",
"_id": "OCScKmIBKBDaRuGPyr42",
"_score": 1,
"_source": {
"demoId": "2",
"demoName": "j"
}
}
]
}
}
}
]
}
}
}

dadoonet · March 17, 2018, 9:34am

Out of curiosity why you are not calling the search(SearchRequest) method? It gives you back a SearchResponse object (including aggs)

Monte_A · March 17, 2018, 9:53am

We do so in our application. However, I am writing junit test and want to verify that the business logic is correct, hence I want to interrogate the SearchResponse object.
If there is better way to write unit test I would be more then happy to hear.

dadoonet · March 17, 2018, 11:15am

As far as I can read the code

github.com

elastic/elasticsearch/blob/6.x/server/src/main/java/org/elasticsearch/action/search/SearchResponse.java#L245-L249


public static SearchResponse fromXContent(XContentParser parser) throws IOException {
    ensureExpectedToken(Token.START_OBJECT, parser.nextToken(), parser::getTokenLocation);
    parser.nextToken();
    return innerFromXContent(parser);
}

It looks like innerFromXContent is also parsing the aggs part.

github.com

elastic/elasticsearch/blob/6.x/server/src/main/java/org/elasticsearch/action/search/SearchResponse.java#L288-L289


} else if (Aggregations.AGGREGATIONS_FIELD.equals(currentFieldName)) {
    aggs = Aggregations.fromXContent(parser);

This is calling:

github.com

elastic/elasticsearch/blob/6.x/server/src/main/java/org/elasticsearch/search/aggregations/Aggregations.java#L134-L152


public static Aggregations fromXContent(XContentParser parser) throws IOException {
    final List<Aggregation> aggregations = new ArrayList<>();
    XContentParser.Token token;
    while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) {
        if (token == XContentParser.Token.START_OBJECT) {
            SetOnce<Aggregation> typedAgg = new SetOnce<>();
            String currentField = parser.currentName();
            parseTypedKeysObject(parser, Aggregation.TYPED_KEYS_DELIMITER, Aggregation.class, typedAgg::set);
            if (typedAgg.get() != null) {
                aggregations.add(typedAgg.get());
            } else {
                throw new ParsingException(parser.getTokenLocation(),
                        String.format(Locale.ROOT, "Could not parse aggregation keyed as [%s]", currentField));
            }
        }
    }
    return new Aggregations(aggregations);
}
}

Behind the scene this needs to know what kind of aggregation you asked for, what is the java type to use. The magic is happening here:

github.com

elastic/elasticsearch/blob/6.x/server/src/main/java/org/elasticsearch/common/xcontent/XContentParserUtils.java#L122-L161


/**
 * This method expects that the current field name is the concatenation of a type, a delimiter and a name
 * (ex: terms#foo where "terms" refers to the type of a registered {@link NamedXContentRegistry.Entry},
 * "#" is the delimiter and "foo" the name of the object to parse).
 *
 * It also expected that following this field name is either an Object or an array xContent structure and
 * the cursor points to the start token of this structure.
 *
 * The method splits the field's name to extract the type and name and then parses the object
 * using the {@link XContentParser#namedObject(Class, String, Object)} method.
 *
 * @param parser      the current {@link XContentParser}
 * @param delimiter   the delimiter to use to splits the field's name
 * @param objectClass the object class of the object to parse
 * @param consumer    something to consume the parsed object
 * @param <T>         the type of the object to parse
 * @throws IOException if anything went wrong during parsing or if the type or name cannot be derived
 *                     from the field's name
 * @throws ParsingException if the parser isn't positioned on either START_OBJECT or START_ARRAY at the beginning
 */

This file has been truncated. show original

So you need to add a prefix like terms# if you are getting back a terms aggregation.

terms#foo as the javadoc example says.

I hope this can help.

Monte_A · March 17, 2018, 12:19pm

Thank you for your help. Behind the scene magic explanation helped me to get working solution.

Will post it after a little bit more testing.

Monte_A · March 18, 2018, 3:14pm

Working solution for someone that might have the same issue.

Elasticsearch response with aggregation example:

gist.github.com

https://gist.github.com/anonymous/cd2de1ccdedf072f40b69b9094afa05e

elasticsearch_response.json

{
  "took": 16,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {

This file has been truncated. show original

Helper method to create NamedXContentRegistry

public static List<NamedXContentRegistry.Entry> getDefaultNamedXContents() {
    Map<String, ContextParser<Object, ? extends Aggregation>> map = new HashMap<>();
    map.put(TopHitsAggregationBuilder.NAME, (p, c) -> ParsedTopHits.fromXContent(p, (String) c));
    map.put(StringTerms.NAME, (p, c) -> ParsedStringTerms.fromXContent(p, (String) c));
    List<NamedXContentRegistry.Entry> entries = map.entrySet().stream()
            .map(entry -> new NamedXContentRegistry.Entry(Aggregation.class, new ParseField(entry.getKey()), entry.getValue()))
            .collect(Collectors.toList());
  return entries;
}

Method to get SearchResponse from String/Json

public static SearchResponse getSearchResponseFromJson(String jsonResponse){
    try {
        NamedXContentRegistry registry = new NamedXContentRegistry(DashboardCuke.getDefaultNamedXContents());
        XContentParser parser = JsonXContent.jsonXContent.createParser(registry, jsonResponse);
        return SearchResponse.fromXContent(parser);
    } catch (IOException e) {
        System.out.println("exception " + e);
    }catch (Exception e){
        System.out.println("exception " + e);
    }
    return new SearchResponse();
}

system · April 15, 2018, 3:14pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.