Using Java High Level Rest client does not auto-retry Bulk requests

Elastic Stack Elasticsearch
1

This is possibly an intentional change, but there is an inconsistency in behavior when switching to using the Java High Level Rest client from the TransportClient in the case of handling of Bulk requests.

Description of Issue
If the BulkProcessor is made to use the High Level Rest Client to issue requests, it is unable to issue retries even though it passes through the Retry handler's canRetry logic

Steps to reproduce
(I can provide a repository which contains all this):

  1. Create an index in ES

  2. Configure the thread_pool to limit the bulk requests and make more likely a rejection. In my case I used a very small queue size of 2

  3. Create a BulkProcessor which submits requests using TransportClient

     BulkProcessor.builder(client, listener);

  4. Submit requests which result in rejection. You will find that the resulting BulkResponse does not contain failures, however querying the /_cat/thread_pool will show rejections, and the document count should have went up based on the total submitted

  5. Create a BulkProcessor which submits requests using the High Level Rest client's client.bulkAsyncmethod:

      BulkProcessor.Builder builder(BulkProcessor.Listener listener) {
         return BulkProcessor.builder((bulkRequest, actionListener) -> {
             
             client.bulkAsync(bulkRequest, actionListener, new Header[0]);
         }, listener);
     }

  6. Submit requests at a rate to create rejections.

  7. Perform the same set of inserts, you will find that the BulkResponse contains failures and the individual Failure objects have an ElasticsearchException which contains "type=es_rejected_execution_exception"

Expected Behavior
Failed bulk requests (from rate limiting) should get retried using the BackoffPolicy in the same way that TransportClient ones are.

It seems possible that the BulkProcessor is not currently intended to be used with the High Level Rest client, as the calls have to manually be made (versus the TransportClient-native builder method). However as the High Level Rest client's bulkAsync method produces BulkResponses, however this usage is demonstrated in the documentation at: https://www.elastic.co/guide/en/elasticsearch/client/java-rest/master/java-rest-high-document-bulk.html

Hypothesis of Root Issue
It seems that the issue stems from the change in how ElasticsearchException gets unpacked from the response. In the TransportClient version, a typed exception (such as EsRejectedExecutionExcption) gets created, and downstream logic is applied using that. However in the HLR version, it ends up as an ElasticsearchException which simply has a detail message containing a string with the "type".

1 Like
2

Thanks Joshua for your detailed analysis, this sounds like a bug to me. And the root is most likely what you have identified, exceptions are treated differently when using the high-level REST client as they are serialized in json format. Would you mind opening an issue in our github repo?

Thanks again
Luca

3

Thanks Luca! I've created a bug report (mostly copying this discussion) at: https://github.com/elastic/elasticsearch/issues/28885

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.