# Failure with sorts and new co.elastic.clients:elasticsearch-java client

**URL:** <https://discuss.elastic.co/t/failure-with-sorts-and-new-co-elastic-clients-elasticsearch-java-client/291511>\
**Category:** Elasticsearch\
**Created:** [December 11, 2021, 4:13pm UTC](https://discuss.elastic.co/t/failure-with-sorts-and-new-co-elastic-clients-elasticsearch-java-client/291511 "2021-12-11T16:13:51Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![ilgrosso](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ilgrosso/32/94485_2.png) [@ilgrosso](https://discuss.elastic.co/u/ilgrosso)\
**Post date:** [December 11, 2021, 4:13pm UTC](https://discuss.elastic.co/t/failure-with-sorts-and-new-co-elastic-clients-elasticsearch-java-client/291511/1 "2021-12-11T16:13:51Z")

</div>

I am upgrading from Java High-level REST client 7.15 to the new Java client.  
I am almost done, it seems only one issue is left.

I am currently using bare `Map<String, Object>` for documents, hence my index requests look like as follows:

```auto
        String index = ...:
        Map<String, Object> document = ...;
        IndexRequest<Map<String, Object>> request = new IndexRequest.Builder<Map<String, Object>>().
                index(index).
                id(event.getAny().getKey()).
                document(document).
                build();
        IndexResponse response = client.index(request);

```

My search requests are instead something like as:

```auto
        SearchRequest request = new SearchRequest.Builder().
                index(index).
                searchType(SearchType.QueryThenFetch).
                query(getQuery(adminRealms, cond, kind)).
                from(itemsPerPage * (page <= 0 ? 0 : page - 1)).
                size(itemsPerPage < 0 ? elasticsearchUtils.getIndexMaxResultWindow() : itemsPerPage).
                sort(sortBuilders(kind, orderBy)).
                build();

        @SuppressWarnings("rawtypes")
        List<Hit<Map>> esResult = null;
        try {
            esResult = client.search(request, Map.class).hits().hits();
        } catch (Exception e) {
            LOG.error("While searching in Elasticsearch", e);
        }

```

All of pre-existing test cases are still working, with exception of a couple which have in common the fact of requesting to sort on field `userOwner` that, being not mandatory, is not present in all `Map<String, Object>` documents.

When I narrow down the failing test cases to the documents having value of `userOwner`, all is working.

Any thought? TIA.

---

<div class="post-metadata">

**Author:** ![ilgrosso](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ilgrosso/32/94485_2.png) [@ilgrosso](https://discuss.elastic.co/u/ilgrosso)\
**Post date:** [December 11, 2021, 4:15pm UTC](https://discuss.elastic.co/t/failure-with-sorts-and-new-co-elastic-clients-elasticsearch-java-client/291511/2 "2021-12-11T16:15:08Z")

</div>

Forgot to add the stacktrace:

```auto
co.elastic.clients.json.UnexpectedJsonEventException: Unexpected JSON event 'VALUE_NULL' instead of '[KEY_NAME, VALUE_STRING, VALUE_NUMBER, VALUE_TRUE, VALUE_FALSE]'
        at co.elastic.clients.json.JsonpUtils.ensureAccepts(JsonpUtils.java:68) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.json.JsonpDeserializerBase$ArrayDeserializer.deserialize(JsonpDeserializerBase.java:315) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.json.JsonpDeserializerBase$ArrayDeserializer.deserialize(JsonpDeserializerBase.java:285) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:72) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:176) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:137) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:85) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.json.JsonpDeserializerBase$ArrayDeserializer.deserialize(JsonpDeserializerBase.java:316) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.json.JsonpDeserializerBase$ArrayDeserializer.deserialize(JsonpDeserializerBase.java:285) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:72) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:176) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:137) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:72) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:176) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:137) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.transport.endpoints.EndpointWithResponseMapperAttr$1.deserialize(EndpointWithResponseMapperAttr.java:56) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.transport.rest_client.RestClientTransport.decodeResponse(RestClientTransport.java:325) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.transport.rest_client.RestClientTransport.getHighLevelResponse(RestClientTransport.java:291) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:144) ~[elasticsearch-java-7.16.0.jar:?]
        at co.elastic.clients.elasticsearch.ElasticsearchClient.search(ElasticsearchClient.java:1487) ~[elasticsearch-java-7.16.0.jar:?]

```

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [December 11, 2021, 4:53pm UTC](https://discuss.elastic.co/t/failure-with-sorts-and-new-co-elastic-clients-elasticsearch-java-client/291511/3 "2021-12-11T16:53:26Z")

</div>

@swallez is it an issue or the expected behavior for the new client?

---

<div class="post-metadata">

**Author:** ![ilgrosso](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ilgrosso/32/94485_2.png) [@ilgrosso](https://discuss.elastic.co/u/ilgrosso)\
**Post date:** [December 13, 2021, 9:40am UTC](https://discuss.elastic.co/t/failure-with-sorts-and-new-co-elastic-clients-elasticsearch-java-client/291511/4 "2021-12-13T09:40:34Z")

</div>

Ref: [Upgrading Elasticsearch by ilgrosso · Pull Request #298 · apache/syncope · GitHub](https://github.com/apache/syncope/pull/298)

---

<div class="post-metadata">

**Author:** ![swallez](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/swallez/32/11972_2.png) [@swallez](https://discuss.elastic.co/u/swallez)\
**Post date:** [December 14, 2021, 3:37pm UTC](https://discuss.elastic.co/t/failure-with-sorts-and-new-co-elastic-clients-elasticsearch-java-client/291511/5 "2021-12-14T15:37:17Z")

</div>

The stacktrace is hard to decypher (we'll work on improving that). Can you capture the response body? Also, what is the contents of `sortBuilders(kind, orderBy)`?

---

<div class="post-metadata">

**Author:** ![ilgrosso](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ilgrosso/32/94485_2.png) [@ilgrosso](https://discuss.elastic.co/u/ilgrosso)\
**Post date:** [December 14, 2021, 3:59pm UTC](https://discuss.elastic.co/t/failure-with-sorts-and-new-co-elastic-clients-elasticsearch-java-client/291511/6 "2021-12-14T15:59:13Z")

</div>

@swallez thanks for your answer.

> Can you capture the response body?

What's the easiest way to capture the response body?

> Also, what is the contents of `sortBuilders(kind, orderBy)` ?

It's

```auto
List.of(new SortOptions.Builder().field(
                        new FieldSort.Builder().
                                field("userOwner").
                                order(SortOrder.Desc).
                                build()).
                        build());

```

I've just tested with 7.16.1, getting the same results.

Regards.

---

<div class="post-metadata">

**Author:** ![swallez](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/swallez/32/11972_2.png) [@swallez](https://discuss.elastic.co/u/swallez)\
**Post date:** [December 14, 2021, 5:10pm UTC](https://discuss.elastic.co/t/failure-with-sorts-and-new-co-elastic-clients-elasticsearch-java-client/291511/7 "2021-12-14T17:10:01Z")

</div>

> What's the easiest way to capture the response body?

There are two main ways:

- set a breakpoint at the beginning of `RestClientTransport.decodeResponse` and inspect `new String(clientResp.getEntity().getContent().readAllBytes())`
- setup a debugging proxy between your application and the Elastiscsearch server, e.g. [Postman](https://learning.postman.com/docs/sending-requests/capturing-request-data/capturing-http-requests/) or [CharlesProxy](https://www.charlesproxy.com/).

---

<div class="post-metadata">

**Author:** ![ilgrosso](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ilgrosso/32/94485_2.png) [@ilgrosso](https://discuss.elastic.co/u/ilgrosso)\
**Post date:** [December 15, 2021, 8:00am UTC](https://discuss.elastic.co/t/failure-with-sorts-and-new-co-elastic-clients-elasticsearch-java-client/291511/8 "2021-12-15T08:00:05Z")

</div>

Hi @swallez here are:

- the request body:

```auto
{
  "from": 0,
  "query": {
    "exists": {
      "field": "id"
    }
  },
  "size": 25,
  "sort": [
    {
      "userOwner": {
        "order": "desc"
      }
    }
  ]
}

```

- the response body: [{ "took": 9, "timed\_out": false, "\_shards": { "total": 1, "su - Pastebin.com](https://pastebin.com/VBEHtVxq)

I'd bet that problems are coming from

```auto
        "sort": [
          null
        ]

```

in the response body.

---

<div class="post-metadata">

**Author:** ![swallez](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/swallez/32/11972_2.png) [@swallez](https://discuss.elastic.co/u/swallez)\
**Post date:** [December 15, 2021, 6:01pm UTC](https://discuss.elastic.co/t/failure-with-sorts-and-new-co-elastic-clients-elasticsearch-java-client/291511/9 "2021-12-15T18:01:01Z")

</div>

@ilgrosso I reproduced the issue locally and it's indeed caused by `"sort": [null]`.

I opened [null value in SearchResponse.hits.hits.sort causes parsing failure · Issue #66 · elastic/elasticsearch-java · GitHub](https://github.com/elastic/elasticsearch-java/issues/66) to track this.

---

<div class="post-metadata">

**Author:** ![ilgrosso](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ilgrosso/32/94485_2.png) [@ilgrosso](https://discuss.elastic.co/u/ilgrosso)\
**Post date:** [December 16, 2021, 7:46am UTC](https://discuss.elastic.co/t/failure-with-sorts-and-new-co-elastic-clients-elasticsearch-java-client/291511/10 "2021-12-16T07:46:54Z")

</div>

Thank you @swallez !  
Shall we expected the fix with 7.16.2?

---

<div class="post-metadata">

**Author:** ![swallez](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/swallez/32/11972_2.png) [@swallez](https://discuss.elastic.co/u/swallez)\
**Post date:** [December 16, 2021, 6:27pm UTC](https://discuss.elastic.co/t/failure-with-sorts-and-new-co-elastic-clients-elasticsearch-java-client/291511/11 "2021-12-16T18:27:15Z")

</div>

@ilgrosso It has been fixed in [Accept null values in arrays by swallez · Pull Request #68 · elastic/elasticsearch-java · GitHub](https://github.com/elastic/elasticsearch-java/pull/68) and will be in the next release.

---

<div class="post-metadata">

**Author:** ![ilgrosso](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ilgrosso/32/94485_2.png) [@ilgrosso](https://discuss.elastic.co/u/ilgrosso)\
**Post date:** [December 17, 2021, 9:47am UTC](https://discuss.elastic.co/t/failure-with-sorts-and-new-co-elastic-clients-elasticsearch-java-client/291511/12 "2021-12-17T09:47:22Z")

</div>

Thanks @swallez .

I did try the current 7.16.2-SNAPSHOT but got the same `"sort": [null]` response: possibly too early to get the updated SNAPSHOT?

Furthermore, I am having troubles with transitive dependencies not being correctly picked up, as - in particular `jakarta.json-api` which is incorrectly taken in its 1.1.6 version, rather than 2.0.1 as I am reading from github.

Thanks for your support.

---

<div class="post-metadata">

**Author:** ![ilgrosso](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ilgrosso/32/94485_2.png) [@ilgrosso](https://discuss.elastic.co/u/ilgrosso)\
**Post date:** [December 20, 2021, 7:42am UTC](https://discuss.elastic.co/t/failure-with-sorts-and-new-co-elastic-clients-elasticsearch-java-client/291511/13 "2021-12-20T07:42:21Z")

</div>

With `7.16.2` I am glad to confirm that everything works, thanks @swallez !

I also have to confirm that there is something messing with transitive dependencies, since without adding

```auto
<dependency>
  <groupId>jakarta.json</groupId>
  <artifactId>jakarta.json-api</artifactId>
  <version>2.0.1</version>
</dependency>

```

I receive `ClassNotFoundException` for `jakarta/json/JsonException`.

---

<div class="post-metadata">

**Author:** ![ilgrosso](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ilgrosso/32/94485_2.png) [@ilgrosso](https://discuss.elastic.co/u/ilgrosso)\
**Post date:** [December 20, 2021, 4:53pm UTC](https://discuss.elastic.co/t/failure-with-sorts-and-new-co-elastic-clients-elasticsearch-java-client/291511/14 "2021-12-20T16:53:14Z")

</div>

@swallez the transitive dependency issue seems to be due to Spring Boot, I did work around that.

Thanks for your support!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 17, 2022, 4:53pm UTC](https://discuss.elastic.co/t/failure-with-sorts-and-new-co-elastic-clients-elasticsearch-java-client/291511/15 "2022-01-17T16:53:22Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.