Failure with sorts and new co.elastic.clients:elasticsearch-java client

ilgrosso · December 11, 2021, 4:13pm

I am upgrading from Java High-level REST client 7.15 to the new Java client.
I am almost done, it seems only one issue is left.

I am currently using bare Map<String, Object> for documents, hence my index requests look like as follows:

        String index = ...:
        Map<String, Object> document = ...;
        IndexRequest<Map<String, Object>> request = new IndexRequest.Builder<Map<String, Object>>().
                index(index).
                id(event.getAny().getKey()).
                document(document).
                build();
        IndexResponse response = client.index(request);

My search requests are instead something like as:

        SearchRequest request = new SearchRequest.Builder().
                index(index).
                searchType(SearchType.QueryThenFetch).
                query(getQuery(adminRealms, cond, kind)).
                from(itemsPerPage * (page <= 0 ? 0 : page - 1)).
                size(itemsPerPage < 0 ? elasticsearchUtils.getIndexMaxResultWindow() : itemsPerPage).
                sort(sortBuilders(kind, orderBy)).
                build();

        @SuppressWarnings("rawtypes")
        List<Hit<Map>> esResult = null;
        try {
            esResult = client.search(request, Map.class).hits().hits();
        } catch (Exception e) {
            LOG.error("While searching in Elasticsearch", e);
        }

All of pre-existing test cases are still working, with exception of a couple which have in common the fact of requesting to sort on field userOwner that, being not mandatory, is not present in all Map<String, Object> documents.

When I narrow down the failing test cases to the documents having value of userOwner, all is working.

Any thought? TIA.

ilgrosso · December 11, 2021, 4:15pm

Forgot to add the stacktrace:

co.elastic.clients.json.UnexpectedJsonEventException: Unexpected JSON event 'VALUE_NULL' instead of '[KEY_NAME, VALUE_STRING, VALUE_NUMBER, VALUE_TRUE, VALUE_FALSE]'
        at co.elastic.clients.json.JsonpUtils.ensureAccepts(JsonpUtils.java:68) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.json.JsonpDeserializerBase$ArrayDeserializer.deserialize(JsonpDeserializerBase.java:315) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.json.JsonpDeserializerBase$ArrayDeserializer.deserialize(JsonpDeserializerBase.java:285) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:72) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:176) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:137) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:85) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.json.JsonpDeserializerBase$ArrayDeserializer.deserialize(JsonpDeserializerBase.java:316) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.json.JsonpDeserializerBase$ArrayDeserializer.deserialize(JsonpDeserializerBase.java:285) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:72) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:176) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:137) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:72) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:176) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:137) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.transport.endpoints.EndpointWithResponseMapperAttr$1.deserialize(EndpointWithResponseMapperAttr.java:56) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.transport.rest_client.RestClientTransport.decodeResponse(RestClientTransport.java:325) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.transport.rest_client.RestClientTransport.getHighLevelResponse(RestClientTransport.java:291) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:144) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.elasticsearch.ElasticsearchClient.search(ElasticsearchClient.java:1487) ~[elasticsearch-java-7.16.0.jar:?]

dadoonet · December 11, 2021, 4:53pm

@swallez is it an issue or the expected behavior for the new client?

ilgrosso · December 13, 2021, 9:40am

Ref: Upgrading Elasticsearch by ilgrosso · Pull Request #298 · apache/syncope · GitHub

swallez · December 14, 2021, 3:37pm

The stacktrace is hard to decypher (we'll work on improving that). Can you capture the response body? Also, what is the contents of sortBuilders(kind, orderBy)?

ilgrosso · December 14, 2021, 3:59pm

@swallez thanks for your answer.

Can you capture the response body?

What's the easiest way to capture the response body?

Also, what is the contents of sortBuilders(kind, orderBy) ?

It's

List.of(new SortOptions.Builder().field(
                        new FieldSort.Builder().
                                field("userOwner").
                                order(SortOrder.Desc).
                                build()).
                        build());

I've just tested with 7.16.1, getting the same results.

Regards.

swallez · December 14, 2021, 5:10pm

What's the easiest way to capture the response body?

There are two main ways:

set a breakpoint at the beginning of RestClientTransport.decodeResponse and inspect new String(clientResp.getEntity().getContent().readAllBytes())
setup a debugging proxy between your application and the Elastiscsearch server, e.g. Postman or CharlesProxy.

ilgrosso · December 15, 2021, 8:00am

Hi @swallez here are:

the request body:

{
  "from": 0,
  "query": {
    "exists": {
      "field": "id"
    }
  },
  "size": 25,
  "sort": [
    {
      "userOwner": {
        "order": "desc"
      }
    }
  ]
}

the response body: { "took": 9, "timed_out": false, "_shards": { "total": 1, "su - Pastebin.com

I'd bet that problems are coming from

        "sort": [
          null
        ]

in the response body.

swallez · December 15, 2021, 6:01pm

@ilgrosso I reproduced the issue locally and it's indeed caused by "sort": [null].

I opened null value in SearchResponse.hits.hits.sort causes parsing failure · Issue #66 · elastic/elasticsearch-java · GitHub to track this.

ilgrosso · December 16, 2021, 7:46am

Thank you @swallez !
Shall we expected the fix with 7.16.2?

swallez · December 16, 2021, 6:27pm

@ilgrosso It has been fixed in Accept null values in arrays by swallez · Pull Request #68 · elastic/elasticsearch-java · GitHub and will be in the next release.

ilgrosso · December 17, 2021, 9:47am

Thanks @swallez .

I did try the current 7.16.2-SNAPSHOT but got the same "sort": [null] response: possibly too early to get the updated SNAPSHOT?

Furthermore, I am having troubles with transitive dependencies not being correctly picked up, as - in particular jakarta.json-api which is incorrectly taken in its 1.1.6 version, rather than 2.0.1 as I am reading from github.

Thanks for your support.

ilgrosso · December 20, 2021, 7:42am

With 7.16.2 I am glad to confirm that everything works, thanks @swallez !

I also have to confirm that there is something messing with transitive dependencies, since without adding

<dependency>
  <groupId>jakarta.json</groupId>
  <artifactId>jakarta.json-api</artifactId>
  <version>2.0.1</version>
</dependency>

I receive ClassNotFoundException for jakarta/json/JsonException.

ilgrosso · December 20, 2021, 4:53pm

@swallez the transitive dependency issue seems to be due to Spring Boot, I did work around that.

Thanks for your support!

system · January 17, 2022, 4:53pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Sort in Java API client Elasticsearch	1	3456	June 20, 2022
ElasticSearch 8.1.1 Java API - Sort doesn't work Elasticsearch	2	1290	April 26, 2022
Query fails via Java client, successful with elasticsearch-head Elasticsearch	9	717	July 6, 2017
New Elasticsearch 8 Java API - Sort by Elasticsearch	5	4305	September 1, 2022
New Elasticsearch client - Search response cannot be serialized because of null value on sort fields Elasticsearch language-clients	10	1523	October 28, 2022

Failure with sorts and new co.elastic.clients:elasticsearch-java client

Related topics