How to use BulkProcessor in synchronous processing?

Elastic Stack Elasticsearch
1

Hello,

I am trying to figure out how to use BulkProcessor [Bulk API | Java REST Client [7.14] | Elastic] during synchronous processing (bulk method) so that I could write something like this:

    val processor = BulkProcessor.builder(
        { request, _ ->
            client.bulk(
                request,
                RequestOptions.DEFAULT
            )
        })

Unfortunately, documentation only mentions async substitutes.

I know that BulkProcessor has concurrentRequests [BulkProcessor.Builder - elasticsearch 6.0.1 javadoc] property which I could set to 0 but is it enough to mimic synchronous behavior?

Has anyone had a similar problem?

Thank you in advance.

2

The bulk processor has never been synchronous. It has always used a listener.
If you want to use synchronous bulk api use Bulk API | Java REST Client [7.14] | Elastic

3

@dadoonet thank you for the response. BulkProcessor allows me to configure backoffpolicy and bulksize. Is it possible to configure mentioned properties using the synchronous bulk method?

4

Why would you need this as BulkProcessor makes by design asynchronous calls?

5

I also wonder the same thing, but regardless of the "why" it's always simple to turn an asynchronous call into a synchronous one: simply make the async call and then block until the listener completes. For instance, create a new java.util.concurrent.CountDownLatch(1) and a listener that calls CountDownLatch#countDown() on completion, then pass the listener into the async call and block the calling thread by calling CountDownLatch#await() on it.

2 Likes
6

@DavidTurner thank you for your answer. Maybe my question was wrong. I would like to set properties like backoffpolicy and bulksize. I know that I can set them using BulkProcessor (hence my question).

If BulkProcessor is designed for async calls then is there any other way to set mentioned properties using sync calls (bulk method)?

7

The parameters you are asking about are parameters of the bulk processor, so it doesn't really make sense to ask about using them without using the bulk processor. Why not just use the bulk processor (modifying it to work synchronously as I described)? If you don't want that, you'll need to write some code to retry with backoff and to size the bulks according to your needs.

1 Like
8

Ok, thank you. I just thought there is some other solution that I am not aware of.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.